Hi everyone,
I know you planned to see about this maybe at the community meeting, but I fear we can not wait for as long.
Paul; could you please make this your current priority, to explore why the wiki is so slow and what we should do about it?
Best,
On 11/15/2017 02:08 PM, Jonas Oberg wrote:
Hi everyone,
I know you planned to see about this maybe at the community meeting, but I fear we can not wait for as long.
Paul; could you please make this your current priority, to explore why the wiki is so slow and what we should do about it?
Best,
Hi Jonas,
The machine seems to be quite busy. Currently the machine has a load of 12 and apache is really busy...
Furthermore something seems hack the root account on SSH...
Nov 15 19:15:38 scheele sshd[15125]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=58.218.198.150 user=root Nov 15 19:15:38 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2 Nov 15 19:15:41 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2 Nov 15 19:15:44 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2 Nov 15 19:15:48 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2 Nov 15 19:15:51 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2 Nov 15 19:15:55 scheele sshd[15127]: Failed password for root from 218.65.30.122 port 49496 ssh2
Best Regards, Thomas
On 11/15/2017 07:16 PM, Thomas Doczkal wrote:
The machine seems to be quite busy. Currently the machine has a load of 12 and apache is really busy...
Hi ,
current load is back to normal and below 3.5. But maybe it's worth to spend some time in configuration of fail2ban for SSH and HTTP Access. It's unlikely all root access users will lock out at the same time though still possible. Good point to discuss.
Best Regards, Thomas
On Wed, Nov 15, 2017 at 07:16:24PM +0100, Thomas Doczkal wrote:
The machine seems to be quite busy. Currently the machine has a load of 12 and apache is really busy...
I currently suspect, that there is some crash condition in Moin which leads up to this. I do not think this is simple overload, but rather a code path which get triggered on certain occasions. The load decreases when the underlying scripts finally terminate, or get killed off due to connection timeout (and when the user gives up hitting F5).
Furthermore something seems hack the root account on SSH...
This is perfectly normal background radiation for any ssh daemon on the public internet. It is also safe to say, that those attempts will never succeed, because we do not even allow password authentication.
Though we can install fail2ban anyway.
On November 16, 2017 3:08:23 PM GMT+01:00, "Paul Hänsch" paul@fsfe.org wrote:
On Wed, Nov 15, 2017 at 07:16:24PM +0100, Thomas Doczkal wrote:
The machine seems to be quite busy. Currently the machine has a load
of
12 and apache is really busy...
I currently suspect, that there is some crash condition in Moin which leads up to this. I do not think this is simple overload, but rather a code path which get triggered on certain occasions. The load decreases when the underlying scripts finally terminate, or get killed off due to connection timeout (and when the user gives up hitting F5).
Furthermore something seems hack the root account on SSH...
This is perfectly normal background radiation for any ssh daemon on the public internet. It is also safe to say, that those attempts will never succeed, because we do not even allow password authentication.
Though we can install fail2ban anyway.
I'm aware of ssh key access only. Nevertheless can we reduce the load of the server by blocking the IP after to many invalid tries.
Regards Thomas
On Wed, Nov 15, 2017 at 02:08:48PM +0100, Jonas Oberg wrote:
Paul; could you please make this your current priority, to explore why the wiki is so slow and what we should do about it?
Okay then!
I already had a first look into it the other day, and my suspicion rests on some Macros which might generate a lot of load when executed.
Funny thing: I do not experience the lag in everyday use (maybe two or three seconds max, sometimes when editing a page). Maybe you could keep an eye out for the pages and actions that immediately lead to larger delays.
And because it was mentioned somewhere else: I do not believe this has to do with file storage. We only have about one thousand articles and all page operations happen in small subdirectories.
On Thu, Nov 16, 2017 at 02:50:36PM +0100, Paul Hänsch wrote:
On Wed, Nov 15, 2017 at 02:08:48PM +0100, Jonas Oberg wrote:
Paul; could you please make this your current priority, to explore why the wiki is so slow and what we should do about it?
Okay then!
I identified the hitcounter as a possible problem. It is used by the <<Hits>> macro and by the statistics subpage for every page. The only page using the macro was the sandbox of my own user page (that is how I got suspicious the first time). The statistics subpages (Info -> Show "Page hits and edits") seem to be rarely clicked by humans but are regularly indexed by search engines.
- I disabled the statistics pages using a rewrite rule - I disabled the macro by unsetting the read permission of /opt/moin-1.9.8/MoinMoin/macro/Hits.py - Independently from this I installed fail2ban which by default is active for ssh
I had a look into the source code of moin-1.9.9 and the problem does not yet seem fixed there (although the devs seem to be aware that the corresponding code needs an overhaul).
Funny thing: I do not experience the lag in everyday use (maybe two or three seconds max, sometimes when editing a page). Maybe you could keep an eye out for the pages and actions that immediately lead to larger delays.
I will try to keep an eye out too. Please let me know, if my changes did not yet do the trick.