Team-BHP > Shifting gears > Gadgets, Computers & Software
Register New Topics New Posts Top Thanked Team-BHP FAQ


Reply
  Search this Thread
15,058 views
Old 25th October 2023, 22:42   #1
BHPian
 
Join Date: Dec 2020
Location: Chennai
Posts: 52
Thanked: 344 Times
The HomeLab Thread | Self-hosting a variety of software & applications

Ha.!, I've finally found a topic that doesn't exist on Team BHP. So, here's my contribution on HomeLabbing.

The name sounds cool right? One might think it's a reference to the Home made Meth labs they show in Breaking Bad. Alas, it is not that cool (or illegal*).

The HomeLab Thread | Self-hosting a variety of software & applications-168419.jpg

The HomeLab you thought of when you opened this page.

The HomeLab Thread | Self-hosting a variety of software & applications-168420.jpg

The HomeLab, I'm talking about


A HomeLab by it's definition found here(link) is "A server set up in one's home for the purpose of testing various configurations of hardware, operating systems, etc.

While the original term might today, only be relevant to some who deals with computer hardware, such as an embedded systems engineer or a hardware engineer, the broader sense of the term now includes everyday Toms, Dicks and Harrys who use laptops and computers to self-host a variety of softwares and applications, that makes life easier for them.

Eh, what is this self host? i thought you were talking about HomeLab?

Self Hosting is the "art" of hosting softwares and applications on devices you own, rather than using those provided by a tech company.

For example, not so long ago, before the advent of Spotify/ Gaana/ Apple Music, we used to store MP3 tracks on our phones to listen when we're out and about. We were self hosting a "music server".

So why should you be interested in HomeLabbing?

You could be interested in HomeLabbing if you are: (in no particular order)

1. Tired of paying subscriptions to the hundreds of apps that you use, just to watch a single movie and then forget about it.
2. You don't want to pay exorbitant amounts for something you know that a simple computer can do, you just don't have the know how.
3. You chose another profession, but you are interested by computers and coding.
4. You don't trust the FAANG companies with your data.
5. You're bored and searching for a new hobby.
6. You like to support and develop open source projects.
7. You have too many old laptops or computers or phones and don't know what to do with them.
8. Have this dying thirst to explore new stuff.
9. You want to host your own website.
10. Any other reason.

What is in a HomeLab?

At the bare minimum, a PC or a laptop or a Single Board Computer (like the Raspberry Pi). Sometime, even just your phone.

What can you do with a HomeLab?

1. Host your own movies, music and even books.
2. Run backups for all your important files and folders.
3. Store photos on your own system and have open source software help you with facial recognition (basically something like Google Photos, except Google doesn't get the facial recognition data)
4. Have a self-hosted password manager.
5. Run plane-tracking software. (ADS-B, nothing illegal)
6. Run a software defined radio that helps you pick up radio signals around you. (Yes, that means everything )
7. Run software that helps with scanning documents and doing OCR(Optical character Recognition) and storing data in a useful manner.
8. Many many many more.

Last edited by Gannu_1 : 26th October 2023 at 10:43. Reason: Uploading images as attachments instead of hyperlinking (breaking page bounds)
alphamike_1612 is offline   (31) Thanks
Old 26th October 2023, 01:26   #2
BHPian
 
Join Date: Dec 2020
Location: Chennai
Posts: 52
Thanked: 344 Times
re: The HomeLab Thread | Self-hosting a variety of software & applications

Now that we know what a HomeLab is, what's in a HomeLab and what you can do with a HomeLab, let's see what I have in my HomeLab.

My equipment:

1. A Raspberry Pi 4 8GB SBC.

2. A Raspberry Pi Zero 2W.

3. An old HP Laptop

4. A 500 GB 2.5 inch laptop Hard Disk.

5. A 1TB 2.5 inch SATA SSD.

6. A 1 TB nVME SSD.

7. A dual band(2.4GHz/5GHz) router with Gigabit LAN ports.

8. 3 RTL-SDR(Software defined radios) dongles.

9. A 150Mbps broadband connection from ACT.

What I run in it:

All computers run on Linux.
- Both the Pi's use Pi OS, the 4 uses Pi OS Bookworm, the Zero 2W uses Pi OS Bullseye.
- The HP laptop runs on Debian Bookworm.

1. Jellyfin - A software that helps you organize and present movies, series, music and books that you own.

2. Navidrome - A dedicated music server.

3. SMB server - Allowing me to access my files across devices.

4. SDR++ - A software that allows me to use RTL-SDR dongles to listen to what's in the airwaves around me. 2 of the 3 dongles are used here. (One for HF and one for VHF).

5. ADS-B tracking - Ever seen flightradar24 and wondered how the website knows where the planes are? This is how...!! The 3rd SDR dongle is used here.

6. PiHole - An Ad blocker that can be used as a DNS server and also stops trackers.

7. Vaultwarden - A self hosted password manager.

8. UrBackup - A self hosted backup server.

9. Kavita - A self hosted e-book reader.

10. Tailscale - A software that allows you to access devices inside your home that don't have a public facing IP. (Thanks to BHPian deep_bang, for introducing me to this wonderful piece of software)

11. qBittorrent - A torrent manager that I **exclusively** use to download only **Linux images**.

12. Pepperminty Wiki - A self hosted wiki that helps me document the things I do in my HomeLab.

13. Glances - A monitoring software for the devices that are in your HomeLab.

14. Homepage - A Homepage for the HomeLab that presents whatever I have in a orderly sense with the ability to get data from APIs.

15. I also use the Raspberry Pi as a daily drive computer and occasionally edit images.

16. Anything else I might have left out.

Some Pictures:

The HomeLab Thread | Self-hosting a variety of software & applications-edited-pi-4.jpg

The first stepping stone -This is the Raspberry Pi 4. It sits inside a metal enclosure taped to the side of my table.

To the right of the cardboard box is a mechanical keyboard button. This button acts as a Power button for my Raspberry Pi. (To safely shut down and start it)

The mess of wires you see include the wires for two 2.5 inch SATA to USB 3.0 disk enclosures and a powered USB hub, to expand the available ports and more importantly to provide the hard disk power to spin up.


The cardboard box you see, was supposed to act as a UPS for the Pi, when power went out, but due to incompatiability with my SDR dongles, it sits unused. The UPS was made by me as is obvious from the dried hot glue.

Below is the trusty "extension" box that provides power for everything. The hub and Pi are powered by 2 old phone chargers.

What you can't see is the SDR dongle, it's taped to the back of my table and I didn't want to move it.


The HomeLab Thread | Self-hosting a variety of software & applications-edited-hdd.jpg

The underside of the table, where the SSD and HDD enclosures reside along with the powered USB hub. You can also see the USB cable for the dongle on the right, along with the ethernet cable attached directly to the Pi4.


The HomeLab Thread | Self-hosting a variety of software & applications-20231025_230547.jpg

The entire table setup viewed at a distance. On the left, next to the table you can see 2 wires. The loop is the ethernet cable, it runs out of my room, to the roof and back into the house near the router. The other is a coax cable for my SDR dongle. Right now, only the VHF is working. I am yet to decide on HF. On the table top is a Dell Keyboard that I got from a friend. A Logitech mouse that I have been using for the past 5 years and a BenQ monitor(27 inch, I guess) that's 3 years old.

Everything is usually run headless, until I want to install something new, or debug something or like right now, when I want to type a thread on Team BHP that'll otherwise be impossible on my mobile or tab.

Oh, as for the poster, I'm a doctor, so that's there to motivate me to do better. I emphasize on my profession here because I want you to know, it doesn't take a network engineer, hardware engineer or a software engineer to set this up. All you need is some elbow grease, patience and the willingness to fail, fail and try again and google and reddit and forums and the willingness to ask people for help when you're stuck.

The HomeLab Thread | Self-hosting a variety of software & applications-20231025_230420.jpg

The second stone - The Raspberry Pi Zero 2 W. It sits inside a metal enclosure taped to the side of a metal "bero" (aka almirah). You can also see the third SDR dongle connected to it and it's antenna on top of the cupboard. The steel acts like a good ground point, giving me good reception of signals.

The wire on the left taped to the cupboard is the remanent of a Google Home Power cable. This Pi was also supposed to be connected to a UPS, but was not due to the same incompatiability that I faced with the Pi 4.

The HomeLab Thread | Self-hosting a variety of software & applications-edited-hp.jpg

The most recent (or the most old) (re)acquistion - This is a 7 year old HP Laptop that my parents got me when I entered college. It has a 6th gen i5 and a no use paltry 2GB Radeon Graphics card. This laptop was abused by me for an entire 6 years, before it was reassigned to my father's office where it was further abused, and has finally made it back here to be used in my setup. This doesn't need a UPS, because it's built in in the form of a battery.

You can also see an SSD on top of it. This is a 1 TB nVME SSD that acts as the remote backup option for the data in my father's office.


I'll finish this post by explaining my ?weird hardware and software setup. I'm extremely proud of the entire setup because I set it all up on my own, (of course with help from the internet) everything from the hardware, to the software, to the wiring.



The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_005141_chrome.jpg
The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_005201_chrome.jpg

This is the homepage for my HomeLab, it contains almost all web-accessible services in my setup in a single page. This is also the homepage for all my devices. You can see the number of links blocked by Pihole. I have 2 because, if one decides to fail, there is always another.

1. I started my first machine with the Pi 4. It was originally just supposed to be a SMB file share(Server). It was essentially supposed to supplement my phone and tab storage so that I wouldn't have to worry about space issues. (in other words, my personal cloud).


The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_010529_xplore.jpg

The SMB File Servers. Both NAS and Media NAS are on the Pi 4. NAS is the SSD for my essential files, while Media NAS is for the **linux images**.
The app is called X-plore File Manager, it helps me automate my file transfers from my phone to the SMB server.

2. Raspberry Pi's use SD cards for storage. They don't have any inbuilt storage, so if you want lots of storage, it's cheaper, faster and safer to store them on an external disk. To this end, I bought a Crucial 2.5 inch SSD for all my data, and a used 500 GB 2.5 inch Laptop HDD from HGST for my **linux images**. The reasoning being that, my data is irrecoverable, so it deserves the SSD. The **linux images** can be re-downloaded, hence they exist on the used HDD that may give up whenever it wants to.


The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_005328.jpg
SDR++ on my tab, it is connected remotely to the dongle connected on my Pi 4.

3. Very soon, I connected my SDR dongle to it and used it to listen to the air around me. The VHF spectrum involves everything that happens around your city, including but not limited to FM radio, Aircraft radios, Emergency Services, Hotels, Private Security and amateur radio operators, etc. (I am a licensed amateur radio operator and I use mine to listen to FM and the amateur bands, listening to anything other than FM and the amateur operators puts you in a legal gray area. As long as you don't make a scene out of it, you should be fine, but your mileage might vary.)


4. Then followed qBittorrent, the necessity to download multiple **linux images** on a regular basis, simply because I was done searching which website hosted which **linux image**.

5. Obviously, when you want to transfer photos and videos and voice between 2 devices and expect it to happen quickly, WiFi quickly starts lagging behind, forcing you to use wired cables.

The ethernet is a Cat 6 cable runs out of my window, to the roof and back down the other side, into my house to connect to the router. I could have routed it internally using the existing conduits, but I wasn't particularly interested in attempting 100 different routes trying to find the right one without the help of an electrician, so i simply drilled 2 holes and did the job.

I had a crimping tool and the ends were then crimped with the RJ45 connectors. Has worked flawlessly till date. Cat 6 gives you speeds upto 10Gigabit, but both the Pi4 and my Home router are limited by Gigabit connections, and I am more than satisfied.

6. Tailscale - This is a software that uses NAT traversal to connect 2 devices on the internet, that maybe behind CGNAT.

To understand CGNAT, think of a multi block hostel complex. Your phone is the room inside one block of the complex. There are several rooms(devices) inside one block of the complex.

One block of rooms is your house's WiFi router. There are several blocks(several WiFi connections) within the complex.

The entire complex is your Internet Service Provider.

Now only your ISP has a road facing address. When a letter(Team BHP email) comes to your room, the postman(Team BHP) gives it first to the security guard at the entrance of the apartment complex.(Your ISP) The guard then gives it to the security guard of your block(WiFi router), who finally delivers the letter to your room.(Your Phone)

Why can't they simply let you have a public address and let the postman deliver it to your door? It's because there are too many rooms and not enough numbers for every room in every block in every hostel in the world. So instead they do this NAT(Network address translation)

Now your ISP and WiFi act like a guard, preventing your room(phone or laptop) from being accessed by anyone outside the apartment complex. Most of the times, they are there for your safety to protect you from thieves(Hackers), but what if you want to get your girlfriend into the room?(Your files), they still won't let you.

You can bribe the block security guard to let you bring her into your room when she reaches your block(by opening ports on your router), but you can't bribe the guard at the entrance of the complex(Your ISP). So how do you sneak your girlfriend in? You ask her to follow the security guard from the entrance to your room and finally you get access to her.

This is essentially what Tailscale does, it finds a path between 2 devices.
You can read more on their site, it offers an excellent free program.

7. Now all this was setup, but whenever I looked at the stats, I would see a 3% CPU utilization and 600-700 MB out of the 8GB used, so obviously the itch increased and I installed PiHole.
PiHole acts as a network wide ad blocker. It has a list of sites that it blocks access to. It also blocks access to trackers that try to find out how much time you spend on each website. With the help of Tailscale, I can now have ad free experience even when I'm out travelling.

8. Very soon, i wanted to start hosting a ADS-B server, so I got a Pi Zero 2W, but again that was having very less utilization, so I installed Vaultwarden. I have wanted a Password manager for a long time, but couldn't bring myself to cough up the money for a paid one, and was not trusting of the free ones, so when the opportunity for a self hosted one came up, I immediately chose it. Since both of these don't require fast internet access, the Pi Zero 2W happily runs on WiFi.

The HomeLab Thread | Self-hosting a variety of software & applications-20231026_00h56m58s_grim.png
This is Dump1090 running on my Pi Zero 2W. This data is then fed to flightradar24 and other websites, who collect data from thousands of such receivers and display planes for us to see.

9. Then I heard of Navidrome and really liked the project, so downloaded it and use it regularly. I use a companion app called Symfonium to access it on my android devices. it's not free, but it's worth the price you pay.

The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_005940.jpg
This is the companion app Symfonium, it lets you play music from Subsonic/ Jellyfin/ Emby and several other servers. No more cursing spotify for not having a music track you wanted to listen to.

As you can see, all is not fine, image thumbnails for all the songs seem to be missing. I will have to sit and debug this issue and maybe report to the coders who run this, but all this is a part of HomeLabbing and it's fun.

10. The laptop and Backup SSD are part of my father's office. The laptop was returned saying it was too slow to work on, so I swapped out the Windows 10 it had for Debian Bookworm, the reason for choosing Debian being that I was used to Pi OS, which itself is very heavily based on Debian, so that it wouldn't be too new for me.

11. The Backup SSD is the remote backup for my father's office. It used to be taken every month to the office, data copied and brought back home. Now I use UrBackup (Along with Tailscale), a software that makes this possible over the internet. Both the office and house have a 150 Mbps connection with a bandwidth of 3.3 TB, so it's quite easy.

The HomeLab Thread | Self-hosting a variety of software & applications-20231026_01h16m00s_grim.png

UrBackup interface - This allows you to make backups of files directly or as images(commonly called snapshots). The image backups help when you are attacked by viruses that take control over your files and lock down the computer. It also supports incremental backups, that is it only copies files that have changed since the last time you backed up.

12. When the laptop came back into my possession, I wanted to try out Jellyfin. This software is a kind of Netflix. It sees the movies you have inside your computer, asks the internet for some information and displays it to you. It also helps you stream videos at a lower display setting, allowing you to save bandwidth. It is able to do this, by using your CPU and GPU to transcode video.

This won't be possible when you use VLC, because all VLC does is, it streams the file as is and the interface looks dumb. Also VLC uses SMB to access remote shares, which isn't intended for large files that are intended to be streamed over a period of time. It is more intended for downloading the file locally to your system and then seeing it from your local device.


The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_005515_jellyfin.jpg
The Jellyfin interface, not showing inside for obvious reasons.
The keen eyed would have noted that the thumbnail doesn't match the title, this is because Jellyfin excepts songs to be in a folder structure, but Navidrome expects them to be all in one folder with the right metadata. I use Navidrome mostly, so this isn't an issue for me.


13. Have you ever had a problem where you spend hours trying to find the solution to something, execute it perfectly and then when the same problem comes back at a later date, you've forgotten everything and you go back and spend hours trying to figure out the same solution again?


If you don't, you're lucky, unfortunately, I'm not, so I use this personal Wiki called Pepperminty Wiki, where I document whatever I do, so that if I have to repeat it someday, it'll become easier.

The HomeLab Thread | Self-hosting a variety of software & applications-20231026_01h25m13s_grim.png

14. I also read E-books, so for this I use Kavita, it's primarily designed as a comic/manga reader, but it also works as a novel reader.

The HomeLab Thread | Self-hosting a variety of software & applications-screenshot_20231026_013142_chrome.jpg

14. You might have noticed that I talked about the UPS and said that it was incompatiable with my current setup. This is because, powering the Pi from the UPS introduces noise into my SDR, which means, I don't have that great reception anymore, so to avoid this, I currently run it directly off the mains. This is not much of an issue, since power cuts are extremely rare here, but this is a problem that I will eventually overcome. I'm still searching solutions for the same.

Last edited by alphamike_1612 : 26th October 2023 at 02:07.
alphamike_1612 is offline   (29) Thanks
Old 26th October 2023, 01:59   #3
BHPian
 
Join Date: Dec 2020
Location: Chennai
Posts: 52
Thanked: 344 Times
re: The HomeLab Thread | Self-hosting a variety of software & applications

The keen eye among you would have noticed in the above screenshots, that most of them have custom domains, this was achieved with Apache and by generating custom SSL certificates and uploading them to each of my devices. The SSL Certificate can be obtained from an actual certificate authority free of cost, but I didn't want to.


What I'm thinking of implementing in my HomeLab -

1. Speedtracker - it is an application that tests your Internet connection speed and lets you know if it's slower. It's like speedtest.net, except it's automated.

2. Immich - The Google Photos Replacement. This is an excellent app for replacing your Google photos, but it's in active development, so I'm waiting for a stable release. I also don't use Google Photos much, so it's not that big of an issue for me.

3. Seafile - The Google Drive alternative*. It allows you to transfer files easily. You can use Tailscale funnel to setup a publicly accessible website, which you can later share among friends, but right now, I don't have the time to set it up.

4. Ghostfolio - Your selfhosted wealth management tracker. It has a collection of Indian stocks too, but again, I don't have the time to set it up currently.

5. Fava - It's a self hosted budget tracker. Once again, lack of time prevents me from setting it up.


Now that I have flexed enough and more about my setup, let's talk about you setting up yours.

What you need -

1. A computer, preferably Linux, because Linux gives you granular control over everything. Windows and macOS are designed for the everyday user and they tend to maintain some control to prevent 'idiot behind the keyboard' syndrome. Linux does no such thing. 'idiot behind the keyboard' = Kiss your work goodbye. But this also forces you to be careful.

2. Choice of an application. It can be one of the few I listed above, it can be anything from this list. As you can see, it contains a lot of applications, from to-do lists to accounting software.

3. Run it on your system. The first time you run it, you'll probably break something, it's okay.
Reddit, Forums and the internet are your source of help. Don't mind reaching into multiple subreddits, keep posting your issue and there will definitely be that one guy who faced the same issue. Conversely, if you do find the solution to your problem, do mention it in your post.

4. Enjoy HomeLabbing, forget that you had a life outside of it.

A few points -

1. Do not get into HomeLabbing, thinking you will save money. You will initially, like I did, when I setup my own NAS instead of buying one. But then you will start going downhill and sinking money and time. For example, I must have sunk over 12 hours just on getting the homepage to run like I wanted it to.

2. I have not talked about Self-Hosting and piracy in this thread, but there is an entire stack of applications, called the arr stack that deals with this, you will definitely find it useful, if you use torrents extensively.(Psst - ARRR!!! sounds like a pirate,it's related to piracy, get it?)

3. If you work in a company, don't forget to ask your computer admins if you can look through the stuff they are throwing out/replacing. This is something every Homelabber will tell you. One man's trash is another man's gold. This is especially true for thin client PCs.

4. And the most important. If this looks overwhelming, trust me it's not. The intital few steps will be difficult, but it will become easier and it will reward you.

5. Don't be afraid to use Docker. it'll make your life easier. I didn't and now I have to search for source code binaries and deal with it's own set of issues, before I can get a service running, when all I had to do was type in 1 line of code on Docker.

6. Oh, and the power consumption? You're worried about spending too much power and your electricity bill shooting up through the roof? Don't worry, my Pi4 in it's current state, draws around 10-13 watts of power and the Pi Zero around 5-7 watts. The Laptop must consume some 15 W. All in all 35-40 watts of power. Equivalent to running a tubelight or a fan. Or in other words, it consumes around 1 unit of electricity per day.

7. Your profession doesn't matter. I've said this already, but I'll reiterate it. All you need is some elbow grease, patience and the willingness to fail and try again and the ability to use Google to your advantage and ask forums for help. If you come across a post of mine in any of the forums related to the topics here, you'll find me crying for help.

8. Your motivations may differ. For me HomeLabbing was a way to get back into computers and coding. It was, is and always will be a passion of mine. (I regret not being able to choose engineering) As is obvious in my posts, I talk about the complications and solving them, for me rather than the end product, solving these problems itself is therapeutic. You might not face problems if you ape copy what the guide tells you to. I don't (and probably won't) and end up breaking stuff.

I hope this thread series atleast piqued your interest in HomeLabbing and Selfhosting and makes you want to explore further.

Regards,
alphamike-1612

Last edited by alphamike_1612 : 26th October 2023 at 02:13.
alphamike_1612 is offline   (26) Thanks
Old 26th October 2023, 06:01   #4
Team-BHP Support
 
Join Date: Feb 2004
Location: Mumbai
Posts: 17,859
Thanked: 77,098 Times
re: The HomeLab Thread | Self-hosting a variety of software & applications

Thread moved out from the Assembly Line. Thanks for sharing!
Aditya is offline   (2) Thanks
Old 26th October 2023, 08:48   #5
BHPian
 
deep_bang's Avatar
 
Join Date: Mar 2007
Location: Bangalore / Boise
Posts: 888
Thanked: 1,272 Times
re: The HomeLab Thread | Self-hosting a variety of software & applications

Fantastic narration as well as a great initiative.
I have a much smaller setup for photos backup and all our phone backups are on my own drive now.

One issue I am facing is a Google photos alternative that can connect to my Pi at home. While connectivity is not a problem with tailscale, I could not find any great app yet. I am using photomap but it has its own issues and is not opensource for me to go and update/improve it.

Considered writing my own google photos alternative but will try the option you suggested.
deep_bang is online now   (3) Thanks
Old 26th October 2023, 09:38   #6
Senior - BHPian
 
aargee's Avatar
 
Join Date: Jan 2010
Location: TSTN
Posts: 6,237
Thanked: 9,647 Times
re: The HomeLab Thread | Self-hosting a variety of software & applications

Technical IT professional here: Personally, I would spend time on availing a service that is absolutely required & is already available {subscription}

However, appreciate your efforts; very few of us in our profession do this, forget alone someone from a non IT background Hats off Sir
aargee is offline   (9) Thanks
Old 26th October 2023, 14:15   #7
Newbie
 
Join Date: Feb 2023
Location: Hyderabad
Posts: 9
Thanked: 12 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

I am blown by this. I am an IT Guy and I don't understand half of it. Kudos to you for learning and doing this setup with passion. You now have an alternate profession (am sure you don't need it) that would do well.
Crank270 is offline   (1) Thanks
Old 26th October 2023, 14:34   #8
BHPian
 
Join Date: Feb 2020
Location: TN22
Posts: 110
Thanked: 533 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Equipments:
1. Intel i5 4500, 16Gb DDR3, 1x 256GB ssd, 2x 1TB HDD, 3x Gigabit NIC
2. Raspberry pi 3 2Gb running raspbian
3. Raspbery pi 4 8Gb running ubuntu

Use cases:

Home server


The intel server is a baremetal running proxmox ce with a lot of virtualization going on.
1. Sophos - Primary firewall. This takes up 2 of the NIC. ISP is connected to 1 and acts as wan. Lan goes to a switch which is connected to another NIC. These 2 are dedicated to firewall since I dont want to overload the NIC. Pretty much selfexplantory on what sophos does. Handles my home network connections, ads blocking, threat mitigation and static routing for my other containers/VM

2. Nginx Proxy Manager - primary proxy manager that I use to forward requests to target vm/containers based on the uri. I have a whole domain which is pointed to my home IP (static) through cloudflare

3. two instances of mariadb containers - 1 used for my freelancing projects dedicated and 1 for hobby projects or other vm/containers. Basically this is a dumping graveyard with tons of DB and random stuffs

4. one instance of postgres container - used for hobby and home use vm/containers

5. one instance of ubuntu VM - dedicated to ads-b tracking and feeding to flight radar 24. Home made antenna and flightradar 24 ADS-B usb dongle. Pretty tiny vm with just 1 core, 512MB ram

6. bunch of nodejs servers/laravel/wordpress (about 10) built on either ubuntu or centos(or its derivatives) usually for my freelancing. the numbers go up and down as I keep deleting or creating new containers based on requirements. Also not all are running all the time

7. windows 11 vm - used for occasional need of a windows machine. rarely used

The 3rd NIC on my home server connects to switch and all other vm/containers except firewall are assigned this 3rd NIC

Raspberry pi 3


Used to host my pihole (dns ad blocking) but now runs a bunch of monitoring tools

Raspberry pi 4


Used for a AI model detection for one of my new hobby project. Bought it just for higher ram requirement.

Quote:
Originally Posted by alphamike_1612 View Post
3 RTL-SDR(Software defined radios) dongles.
I see from one of the images you have a 1/4 wavelength antenna that seem to be used for flight tracking. You should try build one yourself using a plain old coax cable. I had the same setup as you before but moving to the home-made 1/4 wavelength coax antenna improved reception greatly. Also, probably stick that antenna outside you window and preferably running from your terrace

Last edited by vijai : 26th October 2023 at 14:42.
vijai is offline   (3) Thanks
Old 26th October 2023, 15:47   #9
BHPian
 
Join Date: Aug 2017
Location: Leeds
Posts: 937
Thanked: 2,259 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Thanks for sharing this! A friend spent a month to set his up with a Raspberry Pi and it's something I've thought about for a while now - just not gotten around to it. Going to bookmark this for reference when I do eventually try it myself!
ads11 is offline  
Old 26th October 2023, 17:30   #10
BHPian
 
Join Date: Dec 2019
Location: Roaming in KA
Posts: 77
Thanked: 237 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Equipment:
Raspberry Pi 4 with 2 GB RAM and 256 GB SSD on USB 3 external enclosure - Bought this first to run Pihole & this evolved into.....

Orange Pi 5 with 16 GB RAM and 1TB M.2 SSD
Oracle Cloud ARM Ampere A1 Compute with instance 24 GB RAM (Free)



The Oracle Cloud A1 instance does the heavy lifting.
Orange Pi5 is the local server for everything else.

Both of them run Ubuntu and rest everything else is in Docker Containers.

Some of the most important apps I run:
1. Traefik for the reverse proxy. I have a domain name & use Cloudflare to send the sub-domains to Traefik and further onwards to containers.
2. Watchtower & Duin for updating of the containers (both automatic & manual depending on the criticality)
3. Adguard for ad-block & DNS servers to avoid the crappy ISP ones.
4. Duplicati for back-up (both local & cloud)
5. Home Assistant for home automation
6. Nextcloud for documents & notes
7. TTRSS for RSS server
8. Paperless for storing of scanned and OCR documents
9. Uptime Kuma to monitor the health of all the containers and servers
10. Wireguard server and client for VPN
11. Vaultwarden and its backup software
12. A whole bunch of auxiliary containers for the above programs
13 Tailscale and Wireguard VPN cover the networking

Took me sometime to learn to use Docker and Docker-compose. But now once it is running, needs very little maintenance.

By the way,not even remotely associated with IT industry. I built this for tinkering on my own.
The HomeLab Thread | Self-hosting a variety of software & applications-web-capture_26102023_161832_port.slnonline.net.jpg The HomeLab Thread | Self-hosting a variety of software & applications-web-capture_26102023_162048_port.slnonline.net.jpeg

Quote:
Originally Posted by alphamike_1612 View Post
The keen eye among you would have noticed in the above screenshots, that most of them have custom domains, this was achieved with Apache and by generating custom SSL certificates and uploading them to each of my devices. The SSL Certificate can be obtained from an actual certificate authority free of cost, but I didn't want to.
Traefik, NPM, Caddy etc or any other reverse proxy manager will do this automatically for you. Traefik is the best when working with docker as the entire thing can be configured directly & automatically through labels in the docker-compose file.

Quote:
Originally Posted by aargee View Post
Technical IT professional here: Personally, I would spend time on availing a service that is absolutely required & is already available {subscription}
Agree that somethings like email etc must not self hosted. However:

1. Google & its graveyard - https://killedbygoogle.com/ (Enough said)
2. Many times the services with subscription are half baked with "promises" of improvement.
3. The prices are always increasing.
4. Many of the features suddenly stop working because of bug issues, make you to buy new hardware etc.
5. The service can be stopped many times because the profits from it couldn't enable the CEO to buy his 30th yacht. Repeat 1-5
6. Privacy.

I have been burnt many many times with this & that's why I went with the BYOS (Bring your own Server) route. Throwing money at the problem can only solve certain things.

Last edited by car_addict : 26th October 2023 at 17:59.
car_addict is offline   (2) Thanks
Old 26th October 2023, 20:30   #11
BHPian
 
Join Date: Mar 2021
Location: CCU & PHX
Posts: 46
Thanked: 250 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

How do guys keep your Raspberry pi running cool if it runs 24*7 . I’ve a raspberry pi running Volumio to connect to my ceiling speakers with Airplay and Spotify. However streaming lags if I keep my Pi on for more than 6 hours. I have a heat sink installed.

The HomeLab Thread | Self-hosting a variety of software & applications-img_8185.png

I could only wonder throttling issues with FTP servers etc. Maybe in your application lag is not predominantly visible.
TechSavvy is offline   (1) Thanks
Old 27th October 2023, 09:38   #12
BHPian
 
Join Date: Dec 2020
Location: Chennai
Posts: 52
Thanked: 344 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Quote:
Originally Posted by deep_bang View Post
Fantastic narration as well as a great initiative.
I have a much smaller setup for photos backup and all our phone backups are on my own drive now. Considered writing my own google photos alternative but will try the option you suggested.
Thank you deep_bang. Do share how Immich goes. What is your current Photos setup?
Also, a word of caution, Immich explicitly states to not run it with your main source of Photos. This is the reason for me not trying it. I don't have enough space to take a backup copy and run it on a separate system.

Quote:
Originally Posted by aargee View Post
Technical IT professional here: Personally, I would spend time on availing a service that is absolutely required & is already available {subscription}
Thank you aargee, I absolutely agree. Anything critical or of importance is still dealt with by the companies that specialize in it. The rest, such as media servers are like I said, my way of getting back into coding and feeling the joy I used to when I wrote HTML, JS and C in school. As for the Office backup, a remote backup never used to exist and if it did, was very erratic, so with this,even at the risk of losing some data, it is much more regular.

Quote:
Originally Posted by Crank270 View Post
I am blown by this.
Thank you Crank270!

Quote:
Originally Posted by vijai View Post
Equipments:
1. Intel i5 4500, 16Gb DDR3, 1x 256GB ssd, 2x 1TB HDD, 3x Gigabit NIC
2. Raspberry pi 3 2Gb running raspbian
3. Raspbery pi 4 8Gb running ubuntu
That is a sick setup!!

I am planning on getting a Pi 5 and running OpnSense/PFSense with another NIC attached to the PCI slot for my entire network, but that's a pipe dream away for now.

Quote:
I see from one of the images you have a 1/4 wavelength antenna that seem to be used for flight tracking. You should try build one yourself using a plain old coax cable.
I did try the Coaxial Collinear cable, but for some reason, my reception was worser than with the current quarter wave. I also tried the spider legs version (4 to 6 radials with one centre conductor) and the can version with the same result. As for placing it out of my window, that'll take some time and planning for running Coax from the Pi to the window/roof and is something on my to do list.


Quote:
Originally Posted by ads11 View Post
Going to bookmark this for reference when I do eventually try it myself!
Definitely try it, you will not regret it.!!

Quote:
Originally Posted by car_addict View Post

Orange Pi 5 with 16 GB RAM and 1TB M.2 SSD
Oracle Cloud ARM Ampere A1 Compute with instance 24 GB RAM (Free)
That is a good setup!!

My one gripe with my HomeLab is the lack of Docker. When I was first setting it up, I was concerned about the overhead of Docker over the application and instead chose to install everything bare metal. This has often lead to breaking changes or problems when I want to remove the program, and has often limited my scope, because certain projects don't support building from source. However, every time a project arrives destined for Docker, I tell myself, "Why Docker just for this one application" and find ways to install it bare metal.

About the Oracle cloud free tier, I just set it up a few days ago, I was planning on running a simple website or a VPN that'll negate the need for Tailscale. And maybe try my hand at an E-mail server for absolutely non-critical items, like when my torrents finish downloading.

I chose Hyderabad to be my region and now can't change it and neither do I have the option of choosing ARM. I instead went with an AMD core.

I have a few questions for you:

1. How long have you been using the Cloud free tier and what region are you in?

2. Considering Oracle's history with the "free" tier, how do you continue using them?

3. As far as you know personally, has Oracle decided to up and delete someone's instance? (Other than for obvious reasons like torrenting or acting as a Tor node)

4. Could you please explain more about Duplicati and why you chose it? Have you tired other backup software?


Quote:
Originally Posted by TechSavvy View Post
How do guys keep your Raspberry pi running cool if it runs 24*7.
What temperature is your Pi running at? Mine runs at 45-50C all day long and if I am doing something particularly intensive (like having 30 Firefox tabs open or watch a Youtube video) it goes upto 70C. My only cooling is passive, in the form of a metal case, that along with the reasonably open space available underneath the table keeps it cool enough.
I did try active cooling with a fan, but all of those Chinese fans sound like a rocket engine and honestly, make it very hard to sleep in the same room. Both of my Pi's don't have any active cooling.

I have not noticed any lag for any kind usage. The Pi can comfortably reach upto 80-85C before starting to thermal throttle(at least the Pi 4).

If you would like to know if your Pi is throttled, try running
Quote:
vcgencmd get_throttled
The Raspberry Pi forum has an article explaining what the different modes mean. I'll try to link it here, if I can find it.

Also a general question to everyone here, have any of you managed to get a static IP or atleast a non CGNAT dynamic IP from your ISP? If location is of any importance, I live in Chennai.

I use ACT and I asked for either of the 2, or a native IPV6(which they say is available, but as 6to4 tunnelling), they were unable to offer any of them. They denied static IP saying, I had to be a business if I wanted one and that even then, they would simply open ports on their end and link it with my connection (ie.) still CGNAT, but now with the ISP ports open. I asked in several different ways, including saying I wanted to host a minecraft server, but was denied every instance.
alphamike_1612 is offline   (1) Thanks
Old 27th October 2023, 10:09   #13
BHPian
 
Join Date: Feb 2020
Location: TN22
Posts: 110
Thanked: 533 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Quote:
Originally Posted by TechSavvy View Post
How do guys keep your Raspberry pi running cool if it runs 24*7 . I’ve a raspberry pi running Volumio to connect to my ceiling speakers with Airplay and Spotify. However streaming lags if I keep my Pi on for more than 6 hours. I have a heat sink installed.
Seeing your picture, it seems like the pi is inside some drawer? If so, there would be literally no airflow to get rid of the heat. Place it a better ventilated place. I have mine on top of my home server's top exhaust fan which is 120mm. Since my server has no gpu and 3 intake 120mm, there is decent exhaust pressure on the exhaust fan and since there is no gpu in the server, the exhaust air isnt usually hot which is directed over the pi. I havent seen it throttle so far this way.

Also, something to note is, throttling might not just be because of thermals. It could very well be because the pi is underpowered (if you run it from a pc usb port or some usb hub, it definitely is!). Pi prints a message to console while it is underpowered something in line with "Under voltage detected".
Quote:
Originally Posted by alphamike_1612 View Post

That is a sick setup!!

I am planning on getting a Pi 5 and running OpnSense/PFSense with another NIC attached to the PCI slot for my entire network, but that's a pipe dream away for now.
The Pi wont be capable of handling your firewall + your other SDRs. SDR in itself is CPU resource intensive. Best bet is to get a separate Pi may 4? and use pcie raiser card over usb 3 but your performance will have major hit. USB 3 itself is on a shared pcie lane and then you are maxing it out with a pcie NIC. Another bottleneck would be your pi's cpu it self. At best case, you'll get same performance as a standalone off the shelf router or even worse!

Here's my screenshot of sophos on proxmox having 2 vcpu and 4gb ram on idle pretty much doing nothing. I used to run it on 1 vcpu but it was running at 80% load at times and the struggle of firewall was visible. I was using pfsense until about 6 months back and the loads were pretty much similar

Although my firewall handles about 30 clients at peak
The HomeLab Thread | Self-hosting a variety of software & applications-screen-shot-20231027-10.05.42-am.png

Here's the maximum for comparison. Note how the cpu and memory shoots on load.
The HomeLab Thread | Self-hosting a variety of software & applications-screen-shot-20231027-10.11.17-am.png

Last edited by vijai : 27th October 2023 at 10:12.
vijai is offline   (1) Thanks
Old 27th October 2023, 10:22   #14
Senior - BHPian
 
pranavt's Avatar
 
Join Date: May 2008
Location: Mumbai
Posts: 1,682
Thanked: 641 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Quote:
Originally Posted by alphamike_1612 View Post
I have a few questions for you:

1. How long have you been using the Cloud free tier and what region are you in?

2. Considering Oracle's history with the "free" tier, how do you continue using them?

3. As far as you know personally, has Oracle decided to up and delete someone's instance? (Other than for obvious reasons like torrenting or acting as a Tor node)

4. Could you please explain more about Duplicati and why you chose it? Have you tired other backup software?
Not the person you responded to, but here goes:
1) Been using it for over a year. Both the x86 VM and the ARM VM. Region is Mumbai.

2) Nothing valuable is hosted on the Oracle VM. It could die tomorrow and I'd just have to setup a few containers elsewhere. Most of my software is now hosted on my local server anyway.

3) Not personally, no. But my VM has been shut down twice (powered off) due to "lack of usage". I find that reason a little hard to believe considering the aforementioned containers, plus the fact that it tunnels a few gigs of data everyday through wireguard.

Quote:
Originally Posted by alphamike_1612 View Post
Also a general question to everyone here, have any of you managed to get a static IP or atleast a non CGNAT dynamic IP from your ISP? If location is of any importance, I live in Chennai.
ISP at home was happy to provide a static IP for 1.5k/yr. ISP at my office provides a dynamic ipv4 address. Static IPs are available through that ISP as well, but I've decided to against it, for now.

Quote:
Originally Posted by alphamike_1612 View Post
I use ACT and I asked for either of the 2, or a native IPV6(which they say is available, but as 6to4 tunnelling), they were unable to offer any of them. They denied static IP saying, I had to be a business if I wanted one and that even then, they would simply open ports on their end and link it with my connection (ie.) still CGNAT, but now with the ISP ports open. I asked in several different ways, including saying I wanted to host a minecraft server, but was denied every instance.
Change your ISPs if you can. Airtel's business plans provide a static IP. Jio does as well (but only in their more expensive plans). If you have a choice of ISPs, best to shop around. Local ISPs are happy to provide individual services instead of asking you to subscribe to a bouquet.[/quote]

PS: For homelabbing, Pis never really made sense as the primary server(earlier because they came with 100mbit ethernet and only USB2, and recently due to the cost and lack of availability). You can get refurbed desktops/1LPCs on Amazon for 10-15k that have Intel CPUs 6th Gen or newer, 8-16 gigs of ram and an SSD.

I run multiple game servers (Project Zomboid, Factorio, Minecraft) for me and my friends, the usual *arr stuff, and music, TV and movie streaming via Navidrome/Jellyfin, and a bunch of other random stuff like CyberChef, Fileflows, n8n, a couple of self-coded Discord bots (Python). All this and the CPU barely touches 55C when transcoding for Jellyfin or Fileflows.
pranavt is offline   (2) Thanks
Old 27th October 2023, 10:44   #15
BHPian
 
tilt's Avatar
 
Join Date: Sep 2013
Location: Canada / B'lore
Posts: 802
Thanked: 2,819 Times
Re: The HomeLab Thread | Self-hosting a variety of software & applications

Quote:
Originally Posted by alphamike_1612 View Post
*SNIP*

11. qBittorrent - A torrent manager that I **exclusively** use to download only **Linux images**.*SNIP*
Yeah right! I also read this in yesterday's newspaper, so it must be true

Cheers
tilt is offline   (1) Thanks
Reply

Most Viewed


Copyright ©2000 - 2024, Team-BHP.com
Proudly powered by E2E Networks