I have the station and website back up after an extensive power outage here in part of Sacramento.
Bounced Relays for XCP-ng VM Host Server Updates
I had a plethora of patches to update on my XCP-ng systems and I ended up having to restart the toolstacks on my systems. I started on my slave server restarting my “XCP-ng toolstack” which caused issues as it could not contact my master in the pool due to them talking to different versions of the software packages.
Took me a while to see this was the issue because problems started when I updated the Rocky Linux 9.x system and rebooted it and it would not restart. The Rocky Linux 9.x system was of course running on my slave XCP-ng server. After a while I recalled I needed to restart the XCP-ng hosts server’s toolstack which doesn’t interrupt the VM servers running on it as that was the likely issue . When I restarted the toolstack then suddenly the slave wouldn’t come out of maintenance mode. Tried several toolstack restarts and restarting the XAPI service without luck.
I looked at some documentation and saw I needed to disable HA (High Availability) before restarting toolstack and that didn’t help. I decided to reboot but move over my VMS to the master but couldn’t due to the master being a lower version after my tootstak restart. I finally rebooted and had the same issues and took down some other VM’s I couldn’t control but were still up while the server was in maintenance mode.
After rebooting the slave XCP-ng remained in maintenance mode but with my VMs down of course. I had my ShoutCast and IceCast servers for https://dhoytt.com/snake-ice-radio-blog/now-playing/ on my XCP-ng slave host so no one was able to listen while I was doing this however my web site was till up as that VM system was on the master.
After finally looking through logs I saw that the slave was trying to communicate with the master since they were in the same resource pool. The logs showed the reason the slave would not fully boot was it could not communicate with the master. The networking was fine so I figured it was the version causing them to no communicate.
This is when I decided to gracefully shutdown the web server and other VM serves on the XCP-ng master just in case the server needed to be rebooted and not the toolstack just restarted.
Well the simple XCP-ng master toolstack restart allowed the slave to communicate to it and then come back up. The lesson I need to remember is to do all updates and restarts on the master first or switch over the slave to the master if I do it backwards which is too much unnecessary work.
Back and running with a lot of maintenance and updates needed in my server environment as I have been working tirelessly on getting the rental back up to speed and about 4 other projects that consumed my time this year.
Another Update Windows Music Server
I’m still running on my backup server for music stream relays and I need to update the Windows 2019 Datacenter OS with security updates.
I stepped away from doing the updates of the Fedora Linux relay servers for a few hours since its running so good on the backup system. The relay music server I took from Fedora 34 to Fedora 35 and now I’m taking it to the latest Fedora version, Fedora 38. The update system of Fedora Linux has turned out to be very reliable the last several years.
I have a backup Windows Music server but the system reboots quickly enough I just prefer to wait for the main music server to come back up. That takes less time after simple OS security updates than to sync the second server and start where I left off.
So about 15 minutes from this post time and date I will be rebooting the music server and the stream will be down for 5-10 minutes. The music relay server will be running still on the backup Linux Fedora 38 server with no music.
Updating Relay Server Switching to Backup Brief Music Stream Interruptions
I am updating music relay server up several versions of OS from Fedora 34 to Fedora 38 and moving the streams over to a backup VM\server already at Fedora 38. I have to make sure certificates are in place for SSL change streams from music server to this server and some items in the router and get stunnel sync’d.
Let’s see how smooth this goes. The update will be multi-stage update I have performed on the backup and other servers already.
Brief Music Stream Interruption Updating W2019 OS
Needed to update Windows Server 2019 where my music resides and plays SAM Broadcaster to the latest OS security patches. The Linux systems the IceCast and ShoutCast music relays reside on didn’t get updated at this time.
Back up and streaming as I post this!
Updated SSL Certificates Web Site and Music Streams
My SSL site certificates expired on 05/01/2023 and 0n 05/02/2023 Tuesday I updated the website certificates. I however forgot to update the music relay certificates I run through stunnel I discovered when trying to play the network stream in secure fashion on VLC. Now the music streams for IceCast and ShoutCast can be played securely from my website of if you media player requires that secure streams.
Here’s a list of my streams and bitrates you can place in you media players or click on and it may play if you have a compatible media player or browser:
- ShoutCast SSL Secured Streams:
- IceCast SSL Secured Streams:
- ShoutCast non-SSL Streams:
- IceCast non-SSL Secured Streams:
You can also look on my Now Playing Page https://dhoytt.com/snake-ice-radio-blog/now-playing/ for the following media players to listen. There may be a slight delay before they start playing of several seconds especially for the IceCast relay media players embedded on the site.
Listen 320kbps ShoutCast Listen 192kbps ShoutCast Listen 128kbps ShoutCast Listen 64kbps ShoutCast Listen 320kbps IceCast Listen 160kbps IceCast Listen 128kbps IceCast Listen 64kbps IceCastMusic Streams Down Temporarily
I chose to keep the music streams down while I perform some system maintenance after some infrastructure issues I had earlier this week due to a power fluctuation. Took me a while to come up with a viable workaround while I determined the actual issues. Being under the weather with little sleep earlier this week also delayed the troubleshooting.
I’m currently moving my relay server to a new storage repository after the one it was on was corrupted shared from my FreeNAS system. Strangely enough the relay server seemed to be the only VM not previously affected by the corruption but after a reboot of the storage with a new disk and then bringing it up it had some filesystem corruption. I decided I will migrate it to a temporary new storage repository in my XCP-ng and then bring it up and fix the filesystem.
Will be back streaming soon and have the full story of this crash and the many steps to get back up and how the subsequent in motion reconfiguration and making the infrastructure more resilient at https://dhoytt.com/mainblog/.
Music Interruption Due to Atmospheric River Storm
After a power outage due to Atmospheric River Storms in west and in California my music server was down for a while even after the power came back up. One of the power supplies in my C7000 chassis didn’t allow all of my blade servers to come up right away since it didn’t come online right away. The power supply finally cam up along with the music server on that blade. I’ll need to replace that power supply as that’s the second time this has happened in the recent storms. This also happened New Year’s Eve 2023.
Friday Frenzied Workout Music
Taking a nice measured workout and needed some music to go along with it that’s going to carry me through some crunches, leg raise push ups calisthenics aerobic type workout with treadmill at the end.
Going to include artists like Beyonce, Silk Sonic, Ready For The World, The Neville Brothers, Linkin Park and many others.
Thursday Thrills Mix
Set up the typical mixed genre Snakeice’s House of Beats type mix with a series of songs per genre but with mixed eras for this cold Thursday evening. Towards the end of the night it will all be dedicated to the smoot and mellow sounds.
The type of artists are Sean Paul & Tove Lo, DVSN, Sweet Sensation, Prince, Last Poets, Celtic Woman, Sam Smith and Tony Tribe a very eclectic yet cohesive sound. None of that one genre to listen to mess!
I don’t always post but I’m always throwing in special sets from time to time.
_Prince.jpg)
