Login Register

Downtime of several hours on two mornings this past week

Post Reply
User avatar
matthew1
Site Admin
Posts: 14802
Joined: 14 Sep 2002, 11:03
Year and Model: 850 T5, 1997
Location: Denver, Colorado, US
Has thanked: 2856 times
Been thanked: 1332 times
Contact:

Post by matthew1 »

On the 16th and 19th MVS server was down for several hours both times. Something caused it external to the server's routine functioning. More on that below.

Screenshot 2024-08-20 at 2.49.17 PM.jpg

Both events maxed the memory and CPU, and those events caused the OS to kill the database each time, which sounds crazy but that's a normal chain of events. I restarted the db both times and things were back to normal.

So what was it?

Memory doesn't spike due to magic, or cosmic alignment. Something causes it. I didn't have to look far:

Screenshot 2024-08-20 at 2.36.07 PM.jpg
Screenshot 2024-08-20 at 2.36.07 PM.jpg (74.29 KiB) Viewed 1154 times

Hm. That's a whole lot of users on August 16, the day, hour, and minute the first attack occurred. This strongly points to a DDoS attack. The attack, at ~9400 bots (~30X normal traffic), was weak by industry standards, but MVS is a small server by industry standards, so the effect was server knockout.

Yesterday, after the second event, it was clear this was going to happen again and again so I did a sweeping set of changes (reconfigurations). And lo and behold, as if on cue another attack rolled in... and the server passed. I didn't expect the evening attack, but thanks to whoever cued it up because I got to test my changes. :-D Look at the spike at 18:30... it's just as brutal as the others, but MVS kept ticking, which validated my changes.

Screenshot 2024-08-19 at 11.04.08 PM.jpg

If you want to read more about cyberattacks with very recent data, see this https://radar.cloudflare.com/reports/ddos-2024-q2
Help keep MVS on the web -> click sponsors' links here on MVS when you buy from them.

Also -> Amazon link
. Click that when you go to buy something on Amazon and MVS gets a cut!

1998 V70, no dash lights on

1997 850 T5 [gone] w/ MSD ignition coil, Hallman manual boost controller, injectors, R bumper, OMP strut brace

2004 V70 R [gone]

How to Thank someone for their post

Image

User avatar
Krons
Posts: 1113
Joined: 09 Jan 2022, 05:58
Year and Model: 08S60 05XC90 02S60
Location: Des Moines, IA
Has thanked: 198 times
Been thanked: 215 times

Post by Krons »

I thought it was odd when the site wouldn’t pull up. Clearly it’s nefarious groups who hate DIY repair of Volvos.
08 S602.5T/05 XC902.5T/02 S602.4T
08 C702.5T (sold)
05 S402.4i (RIP, timing belt failure)
The non-Swedes:
25 Mazda MX-5 / 17 Frontier Pro-4X / 17 Ford Focus
17 R1200GS / 15 Versys 1000 / 11 DR-Z400S

User avatar
volvolugnut
Posts: 6502
Joined: 19 Jan 2014, 09:51
Year and Model: 2001 V70
Location: Oklahoma USA
Has thanked: 987 times
Been thanked: 1075 times

Post by volvolugnut »

THEY are just jealous of our Volvos and the MVS community.
Matt, do you have a source for server advice like MVS for Volvo owners?
volvolugnut
The Fleet:
Volvo: 2001 V70 T5, 1986 244DL, 1983 245DL, 1975 245DL, 1959 PV544, multiple Volvo parts cars.
Mercedes: 2001 E320, 1973 280, 1974 280C, 1989 300E, 1988 300TE, 1979 300TD, parts cars.
2009 Smart Passion
Ford: 1977 F350, 1964 F150 (2), 1938 Tudor Sedan
Farmall tractors: 1956 400 Diesel, 1946 A
John Deere 214 lawn tractor about 1977 model
And others.

User avatar
hiroku
Posts: 46
Joined: 28 Jul 2014, 14:48
Year and Model: 2002 V70 T5
Location: Lisbon
Has thanked: 21 times
Been thanked: 9 times

Post by hiroku »

Wonder what's so special about Volvo forums that make them a desirable target for these things.

The local Portuguese Volvo forums are currently a barren wasteland. The last 6 months or so are only posts about bots talking to each other about good-looking females in their area.

The UK Volvo Owner's Club was apparently also attacked repeatedly by Portuguese IPs, which is why I am blocked from accessing it from my home Wi-Fi (I am innocent though, I swear!).

User avatar
matthew1
Site Admin
Posts: 14802
Joined: 14 Sep 2002, 11:03
Year and Model: 850 T5, 1997
Location: Denver, Colorado, US
Has thanked: 2856 times
Been thanked: 1332 times
Contact:

Post by matthew1 »

hiroku wrote: 21 Aug 2024, 06:27 Wonder what's so special about Volvo forums that make them a desirable target for these things.
This chart is particularly chilling to me. What if it is a competitor? How could one forum operator do this to another? I'm not pointing the finger, but data is data.

Screenshot 2024-08-21 at 7.18.02 AM.jpg

That's taken from the link I included at the bottom of my head post.

I read many years ago a phrase that put the Internet into a perspective that's served me well over the years: the Internet is a dark, mysterious place (paraphrased).

MVS has a wide footprint and a longtime (2001, domain name registered 2002) persistence that gives it a certain profile on the web that will pop up on Bad Actor's target lists. That's just the nature of the beast.

So who knows.
Help keep MVS on the web -> click sponsors' links here on MVS when you buy from them.

Also -> Amazon link
. Click that when you go to buy something on Amazon and MVS gets a cut!

1998 V70, no dash lights on

1997 850 T5 [gone] w/ MSD ignition coil, Hallman manual boost controller, injectors, R bumper, OMP strut brace

2004 V70 R [gone]

How to Thank someone for their post

Image

User avatar
BlackBart
Posts: 6725
Joined: 10 Dec 2016, 17:03
Year and Model: 2004 XC70 BlackBetty
Location: Over the far far mountains
Has thanked: 967 times
Been thanked: 921 times

Post by BlackBart »

DDoS = Dedicated Division of Sabotage??
ex-1984 245T wagon
1994 850T5 wagon
2004 XC70 wagon BlackBetty

User avatar
matthew1
Site Admin
Posts: 14802
Joined: 14 Sep 2002, 11:03
Year and Model: 850 T5, 1997
Location: Denver, Colorado, US
Has thanked: 2856 times
Been thanked: 1332 times
Contact:

Post by matthew1 »

Help keep MVS on the web -> click sponsors' links here on MVS when you buy from them.

Also -> Amazon link
. Click that when you go to buy something on Amazon and MVS gets a cut!

1998 V70, no dash lights on

1997 850 T5 [gone] w/ MSD ignition coil, Hallman manual boost controller, injectors, R bumper, OMP strut brace

2004 V70 R [gone]

How to Thank someone for their post

Image

User avatar
abscate
MVS Moderator
Posts: 36147
Joined: 17 Feb 2013, 17:54
Year and Model: 99: V70s S70s,05 V70
Location: Port Jefferson Long Island NY
Has thanked: 1610 times
Been thanked: 4065 times

Post by abscate »

Denial of Service. Basically a traffic jam.
Empty Nester
A Captain in a Sea of Estrogen
1999-V70-T5M56 2005-V70-M56 1999-S70 VW T4 XC90-in-Red
Link to Maintenance record thread

User avatar
matthew1
Site Admin
Posts: 14802
Joined: 14 Sep 2002, 11:03
Year and Model: 850 T5, 1997
Location: Denver, Colorado, US
Has thanked: 2856 times
Been thanked: 1332 times
Contact:

Post by matthew1 »

volvolugnut wrote: 20 Aug 2024, 16:15 THEY are just jealous of our Volvos and the MVS community.
Matt, do you have a source for server advice like MVS for Volvo owners?
volvolugnut
Probably 3/4 time I wind up on https://serverfault.com or one of its sister sites to find answers. https://stackoverflow.com https://superuser.com https://askubuntu.com https://dba.stackexchange.com

My 53-year-old body enjoys my 53-year-old brain enjoying solving problems in the nonphysical world of servers rather than the physical world of cars. Both of these things are identical in concept: problem -> find root cause -> learn how to solve root cause -> solve root cause -> test.
Help keep MVS on the web -> click sponsors' links here on MVS when you buy from them.

Also -> Amazon link
. Click that when you go to buy something on Amazon and MVS gets a cut!

1998 V70, no dash lights on

1997 850 T5 [gone] w/ MSD ignition coil, Hallman manual boost controller, injectors, R bumper, OMP strut brace

2004 V70 R [gone]

How to Thank someone for their post

Image

User avatar
matthew1
Site Admin
Posts: 14802
Joined: 14 Sep 2002, 11:03
Year and Model: 850 T5, 1997
Location: Denver, Colorado, US
Has thanked: 2856 times
Been thanked: 1332 times
Contact:

Post by matthew1 »

Three mornings ago we had another attack, largest yet. The site was probably partly or fully inaccessible -- I don't know I was sleeping -- during this time. The database did not crash, instead the MVS server did the best it could during this ~2 hour timeframe and resumed nominal operation when the attack exhausted itself.

I modified protection measures after this.

I'll reset Most Users Ever now so I can get a high water mark again if another attack rolls in.

Screenshot 2024-08-31 at 1.41.43 PM.jpg
Screenshot 2024-08-31 at 1.41.43 PM.jpg (35.35 KiB) Viewed 1066 times
Help keep MVS on the web -> click sponsors' links here on MVS when you buy from them.

Also -> Amazon link
. Click that when you go to buy something on Amazon and MVS gets a cut!

1998 V70, no dash lights on

1997 850 T5 [gone] w/ MSD ignition coil, Hallman manual boost controller, injectors, R bumper, OMP strut brace

2004 V70 R [gone]

How to Thank someone for their post

Image

Post Reply