RegisterRegister    Log inLog in    SearchSearch   

Post new topic   Reply to topic
 
View previous topic :: View next topic  
Author Message
Kybber



PostPosted: Sat Mar 04, 2006 11:25 am    Post subject: How often does GC check for dead servers? Reply with quote

I've noticed that if a server dies, the log will still keep listing
lines of this sort:
Code:
Sat Mar  4 02:10:20 2006: ** Check Booking Performance: ID 277591 **
Sat Mar  4 02:10:22 2006: SSL >> ACK: PROCPERF cpu=0.0&id=277591&memory=0&pid=18567

every two minutes or so. So instead of trying to restart the
server, GC keeps polling it for resource-usage. This goes on
for about 15 minutes.

Here's a pstree listing of gamecreate running a working BF2
server:
Code:
gamecreate.x86---gamecreate.x86-+-gamecreate.x86---bf2---4*[{bf2}]
                                `-gamecreate.x86

Here's the same pstree after I kill BF2:
Code:
gamecreate.x86---gamecreate.x86-+-gamecreate.x86---bf2
                                `-gamecreate.x86

So the one bf2 process still lingers. It is, however, defunct:
Code:
  PID TTY      STAT   TIME COMMAND
18821 ?        Zs     1:01 [bf2] <defunct>


So my questions are:
1. How often does GC try to restart dead servers?
2. Why does it keep polling for resource usage when the
server is dead? Is this a bug for the Linux-version?

For most games, it doesn't really matter if it takes fifteen
minutes to restart since it happens like every other week.
But for BF2 it's a totally different matter - large BF2 servers
crash more than once per day. Our previous solution which
used Monit to monitor servers would poll servers every
minute and restart them if they were dead. Is it possible to
make GC check the status of servers more frequently?

Edit: BTW, the timestamp for the RSS feed is off by one
hour. The server was restored at 02:16 AM, which is stated
by both the failure report and the RSS feed, but the date-
stamp for the feed says 03:16. This is using UTC+1 for
timezone.
Back to top
View user's profile Send private message
adbot



PostPosted: Thu Mar 31, 2011 3:57 am 

Kybber



PostPosted: Sun Mar 05, 2006 1:09 am    Post subject: Reply with quote

I've done a quick bit of research with my uncle Google:
Aparently a defunct process is the same as a zombie, i.e. a
process whos parent has not yet received any signal that it
is dead.

If I start a BF2 server in a screen session (screen ./start
etc) , then that screen becomes the parent of the BF2
process. If the BF2 process is killed or otherwise crashes, it
sends a signal to the screen process, and the screen exits
immediately. This signal is typically a sigsegv if the server
crashes, but it can also be a sigterm or a sigkill if the server
is stopped with intent.

So it seems to me that GameCreate doesn't monitor for
signals from its children. Is this correct? In my opinion it
would be a great feature for GC to set up signal traps for its
children since that would detect a crashed server immediately,
and the new restarted server could be up and running within
seconds! Very Happy Setting up such traps takes no resources, so
there should be no additional overhead from this.

If using signal traps is not your cup of tea, then it would at
least be appreciated if the interval between checks were
lowered to a minute or even less. To illustrate the effect of
not restarting the server shortly after a crash, here are two
graphs that show the number of players on two different
servers. The first server is monitored by GC and takes 10-
15 minutes to restart after a crash. The second server is
monitored by Monit and takes up to 1 minute to restart:

(query failure detected by GC at 16:55, 21:44 and 12:37)

(crashed at 16:54, 17:36, 20:52 and 14:00)
As can be seen, the first server takes forever to refill since
players decide to join other servers instead of waiting, while
the second server fills up immediately after a crash.


Last edited by Kybber on Sun Mar 05, 2006 1:32 am; edited 2 times in total
Back to top
View user's profile Send private message
Guest



PostPosted: Sun Mar 05, 2006 1:24 am    Post subject: Reply with quote

Or to sum it up in one sentence:

This needs to be addressed quickly, or GC is more or less useless for BF2-servers.
Back to top
Nathan
Mammoth


PostPosted: Mon Mar 06, 2006 4:59 pm    Post subject: Reply with quote

Please let me know if this executable fixes the problem:

http://www.gamecreate.com/~nats/gamecreate.x86

It should prevent the game processes from becoming defunct.
Back to top
View user's profile Send private message
Kybber



PostPosted: Mon Mar 06, 2006 10:03 pm    Post subject: Reply with quote

Well, technically it does what you say. But it doesn't solve
the problem, and brings with it even more serious problems.

When I start a new BF2 server, gamecreate spawns a new
child called gamecreate.x86. This is different from the previous
version which spawned the BF2 process itself, as can be seen
in my pstrees above. Now it looks like this:
Code:
$ pstree 30624
gamecreate.x86(30624)---gamecreate.x86(30625)-+-gamecreate.x86(30626)---gamecreate.x86(31012)
                                              `-gamecreate.x86(30627)

There were only four gamecreate.x86 processes before the
server was started. Process 31012 is new. 31012 is also
the PID reported in gamecreate.log:

Code:
Mon Mar  6 12:40:44 2006: ** Start GameCreate Booking: ID 280060 **
Mon Mar  6 12:40:44 2006: Launching process: /home/server/games/bf2/start.sh +config "configs/16617/serversettings.con" +mapList "configs/16617/maplist.con" +pbPath "configs/16617/pb"  from directory /home/server/games/bf2
Mon Mar  6 12:40:44 2006: SSL >> ACK: START id=280060&pid=31012

However, 31012 is now defunct:
Code:
$ ps 31012
  PID TTY      STAT   TIME COMMAND
31012 ?        Zs     0:00 [gamecreate.x86] <defunct>

Now for the real problem: I am unable to restart the game
server from GC. Here's the log:
Code:
Mon Mar  6 12:45:26 2006: ** Stop GameCreate Booking: ID 280060 **
Mon Mar  6 12:45:27 2006: SSL >> ACK: STOP id=280060
 [20 bytes] - result: 20
Mon Mar  6 12:45:27 2006: Sending PONG, waiting for response
Mon Mar  6 12:45:27 2006: SSL >> ACK: PONG
 [11 bytes] - result: 11
Mon Mar  6 12:46:06 2006: ** Stop GameCreate Booking: ID 280060 **
Mon Mar  6 12:46:12 2006: Waiting for PID: 31012 failed
Mon Mar  6 12:46:12 2006: SSL >> ACK: STOP id=280060
 [20 bytes] - result: 20
Mon Mar  6 12:46:18 2006: ** Create directory: /home/server/games/bf2/pbscreens/Battlefield_2_Server **
Mon Mar  6 12:46:18 2006: ** Create directory: /home/server/games/bf2/logs/280060 **
Mon Mar  6 12:46:18 2006: ** Remove directory: /home/server/games/bf2/mods/bf2/settings **
Mon Mar  6 12:46:18 2006: ** Remove directory: /home/server/games/bf2/mods/bf2/configs **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/configs/16617/maplist.con, size: 512 **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/configs/16617/pb/pb/pbsv.cfg, size: 2589 **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/configs/16617/modmanager.con, size: 5396 **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/admin/admin_rcon16617.cfg, size: 151 **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/configs/16617/adminsettings.con, size: 485 **
Mon Mar  6 12:46:18 2006: ** Got file, name: /home/server/games/bf2/configs/16617/serversettings.con, size: 1684 **
Mon Mar  6 12:46:21 2006: ** Got file, name: /home/server/games/bf2/configs/16617/pb/pb/pbsvlog.cfg, size: 122 **
Mon Mar  6 12:46:21 2006: ** Got file, name: /home/server/games/bf2/configs/16617/actlogpbss.cfg, size: 129 **
Mon Mar  6 12:46:21 2006: ** Got file, name: /home/server/games/bf2/configs/16617/admincommands.con, size: 14472 **
Mon Mar  6 12:46:21 2006: ** Start GameCreate Booking: ID 280060 **
Mon Mar  6 12:46:21 2006: Launching process: /home/server/games/bf2/start.sh +config "configs/16617/serversettings.con" +mapList "configs/16617/maplist.con" +pbPath "configs/16617/pb"  from directory /home/server/games/bf2
Mon Mar  6 12:46:21 2006: SSL >> ACK: START id=280060&pid=31523

First of all, it takes close to a minute from when I issue the
stop command until GC starts the server. However, the
original instance hasn't actually been stopped, so now we
have two BF2 processes running:
Code:
$ ps x | grep bf2
31013 ?        Sl     0:33 /home/server/games/bf2/bin/amd-64/bf2 +config configs/16617/serversettings.con +mapList configs/16617/maplist.con +pbPath configs/16617/pb
31524 ?        Sl     0:23 /home/server/games/bf2/bin/amd-64/bf2 +config configs/16617/serversettings.con +mapList configs/16617/maplist.con +pbPath configs/16617/pb


And back to the original problem: It appears that the time it
takes for the server to restart after I kill the bf2 process is
about the same as before. GC keeps polling the defunct
gamecreate.x86 process every other minute and reporting
0% CPU and RAM usage for 13 minutes before it restarts
the server.
Back to top
View user's profile Send private message
Nathan
Mammoth


PostPosted: Tue Mar 07, 2006 8:32 am    Post subject: Reply with quote

You're right, that change isnt a good one - I'd suggest going back to the normal client as on the website.

I'm going to setup a BF2 linux test server and do some more work on this today.
Back to top
View user's profile Send private message
Kybber



PostPosted: Tue Mar 07, 2006 10:19 am    Post subject: Reply with quote

Thanks, much appreciated. Please let me know if I can do
some testing or otherwise assist you in any way.
Back to top
View user's profile Send private message
Nathan
Mammoth


PostPosted: Tue Mar 07, 2006 11:07 am    Post subject: Reply with quote

Had a bit more of a dig. GameCreate was monitoring its child processes, but was doing so from the wrong thread.

I've fixed that up so the monitoring is from the correct thread, which should mean GameCreate picks up on the dead server in ~15 seconds.

http://www.gamecreate.com/~nats/gamecreate.x86 (still labelled 5.1.1)

I tested this on my own by just doing

kill -SEGV pid

to simulate a server crash, and it seems to be right.
Back to top
View user's profile Send private message
Kybber



PostPosted: Tue Mar 07, 2006 11:57 am    Post subject: Reply with quote

Excellent! I've tested it and it seems to be working perfectly
now. The server is detected as being down within 15
seconds, GC makes sure it is really stopped 15 seconds
after, and the new server instance is started after another
15 seconds.

Great job - thanks, Nathan Smile
Back to top
View user's profile Send private message
Nathan
Mammoth


PostPosted: Tue Mar 07, 2006 2:08 pm    Post subject: Reply with quote

Excellent, I will put a new installer on the site soon.
Back to top
View user's profile Send private message
AntiVirus



PostPosted: Sat Aug 30, 2008 1:44 am    Post subject: Reply with quote

I know its kinda outdated but I have the same problem! I'm starting the BF2 server in gamecreate:

:~# ps aux |grep bf2
root 3063 24.4 0.0 0 0 ? Zs 17:48 0:01 [bf2] <defunct>

Before I was running gamecreate on a separate user, the same happend!

All files have chmod 777... And I changed in gc panel from "Dont use screen" to "Require screen" - still nothing.

If I copy-paste the starting command line from GC to SSH, it works perfectly!

Anyone has got something like this?
Back to top
View user's profile Send private message MSN Messenger
Kybber



PostPosted: Sat Aug 30, 2008 3:53 am    Post subject: Reply with quote

If you do a search for BF2 or BF2142, you'll find
that this is a very common problem which seems
to affect everyone, yet it hasn't been fixed. A
workaround can be found here:
http://www.gamecreate.com/forum/viewtopic.php?p=6198#6198
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic All times are GMT + 10 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Forum powered by phpBB © 2001, 2005 phpBB Group
GameCreate Service Terms | Privacy Policy | © Copyright Mammoth Media 2001-2007
GameCreate™ is a trademark of Mammoth Media Pty Ltd. GameCreate® is a registered trademark in Australia.