Prolonged service outage

Thanks for the explanation Aaron - what a bugger

Should have said what a bugger the "stealer" is

Yeah.

We have worked non stop since xmas eve right over the break to prepare and optimise the platform for high traffic loads, really speeding, toughening up and simplifying all major parts of the platform and it was starting to run extremely well.

Then of course just when things are starting to run smoothly... 🤣

Over the weekend, we provisioned a new enterprise grade web application firewall to now protect the platform from malicious actors and scrapers, bots etc.

All requests to both our web site and our API now have to go through the new kid in town - which will verify/validate all requests against a number of rulesets and decide whether to allow requests through or not before they can hit our platform and consume our resources.

As of this morning, I think it's working.

Note: only naturemapr.org is working at the moment - subdomains like canberra.naturemapr.org are not back up yet.

Thank you, Aaron and team - for your time and patience in explaining what and why things are happening, and using platforms such as Facebook to keep users up to date. Of course, the major 'thank you' is for your expertise and time spent on fixing this major problem, and getting data and the platform protected and up to speed again.(it's very exciting to be able to write and see NM again!!)

I think we're back up and running with new web app firewall in place. Thanks all for your patience and support.

Thanks to all involved in getting things up and running again. John

Thanks for the detailed explanation Aaron, hope we can keep those nasty data thieves out now.

Thanks for restoring CNM so quickly, and so much for a relaxing weekend. Its sad there is so much low life around in this electronic era.

We're definitely not out of the woods yet - the attacks are continuing unfortunately and we're trying to fine tune the new infrastructure to deal with it.

Aaron, What a nightmare for you and the team. I'm a bit stunned that you have been able to get the firewall installed and the system up and running again so quickly, so congrats on that. I had expected it might take a couple of days. I agree it's sad that there are such low life's out there or those perhaps just wanting to show off their hacking skills or worse, deliberately trying to steal data. Thanks heaps to the team for working so tirelessly over the weekend.
Alison

Just for context for those interested in this stuff:

Even right now with new web app firewall turned on, *something* has downloaded almost 100,000 images from us within the past 1 hour (also 60,000 images in the last 30 mins) Or about 33 images per second.

They are fetching these images along with hammering ALL user field guide and user sighting list pages.

We have slowed some of it down, but not all of it.

It is pointing more and more towards it being a distributed attack, because as we deny traffic to sources that are exceeding our new rate limit threshold, more requests are appearing from elsewhere.

I have been worried we accidentally deployed something on Saturday that caused this, but we've retraced all our steps and reverted the changes we did make. These changes were tiny and insignificant. Reverting these changes has also had zero effect and the platform continues to get hammered by something.

We are now suspecting more of a DDOS style attack as opposed to a scraping attack. If you wanted to scrape all NatureMapr content and populate or train your own database or AI model, you would tend to visit a page, rest for a minute, then visit the next page etc. To avoid being detected and to tread lightly on the destination infrastructure so you could get what you want.

Where as this is more of a relentless hammering coming from multiple sources.

Thanks Aaron. I wouldn't be upset if you shut the system down again to protect our data if needed, until it can be fully resolved. Alison

Is what is happening 'illegal', or just a 'cost' of doing business? Donation time?

Wow. This is horrible. I don't understand why someone would use a denial of service attack on NatureMapr. What do they have to gain? Thank you for working so hard trying to stop it Aaron. Great that you managed to install the firewall so quickly. Very annoying that the attack is still getting through. Thank you for keeping us all informed of progress. Katarina

Thanks all. Yes very much part of doing business on the internet these days. We've been very lucky to date that this hasn't happened earlier.

Appreciate the thought @brunonia, but we've made the decision to not accept donations from individuals or community groups any more.

NatureMapr needs to stand on its own sustainably.

Excellent work as ever !!! :)

Hi All,

We've needed to implement a change that will affect the structure of all user profile pages.

To date, we've hosted separate region based versions of everybody's user profile.

For example:

naturemapr.org/MichaelB (all sightings)
canberra.naturemapr.org/MichaelB
sydney.naturemapr.org/MichaelB

Each with a unique web address E.g. Canberra. Sydney. Pilbara.

This essentially creates a massive cross product of page combinations for the system to maintain and host and provides a very large footprint for malicious actors to target because we are essentially hosting 41 different versions of every single user profile page. Not to then mention the pagination that can happen within each one of those.

This is a legacy of the regional structure we started out with and is simply not sustainable to keep doing going forward.

User profile pages will now auto redirect to the main naturemapr.org version, which will be the single, consistent web address and version of everyone's user profile pages.

We understand that some users will miss having a dedicated canberra/michaelb version of their records in the short term. Longer term, we may investigate introducing a way to filter records on your user profile page by collection (some of which are regions) to get this kind of functionality back.

But in the short to medium term, we are killing ourselves with this overly generous architecture and need to drastically reduce our footprint to be able to keep operating the platform sustainably.

Thanks everybody for your support and understanding.