Groovyarcade: Attract mode freezes

[Banane]

Hi everyone,

this is unfortunatley my first post, i came here cause I think itīs f**king great that there is a Groovymame Distro based on arch Linux.
I pray this projekt is still getting better 

I do this thread here to say Hi and also for better visibility.

In Thread " GroovyArcade live-CD New Release 2019 " mmandrea in february had two issues:

a) attract mode hangs / freezes
b) he couldnīt get the onyxarcade themes for attract mode to work ( e.g http://www.onyxarcade.com/flat_blue.html)

Does anyone have an helping update for these topics? For a) I set the attract mode debug level to the highest value, I search in kernel message and /var/log/* and journalctl => just nothing.
Any Idea how to debug this at least ?  .

Greetings from Hamburg / Germany,
Christian

[Doozer]

Wrt to the freezes. Can you check the behavior using ffmpeg? Try to play a video (few minutes) in X with ffplay. It the counters+video stop and the sound plays for few sec, then it is not related to attract but another layer in the OS.

[Banane]

thanks for your reply. I must say the problem is not related to ffmpeg...

I just ordered a new hardware setup and will see if it will return on that one as well...

[Banane]

okay, i bought a brand new ASRock A320M-HDV with a Ryzen 2400G just installed GroovyArcade of the newest ISO file
=> attract mode freezes 

I found this thread here in ukvac-forum, from october 2018, same problem:

http://www.ukvac.com/forum/attractmode-hangs-on-grooveyarcade_topic366595.html

The solution was to switch to Windows 

[Doozer]

okay, i bought a brand new ASRock A320M-HDV with a Ryzen 2400G just installed GroovyArcade of the newest ISO file
=> attract mode freezes 

I made the same assumption and even created a ticket long time ago in attract git. But with some additional investigation I was able to confirm (using ffplay) that the problem is not linked to attractmode.

Kernel regression <4.8 make the phenomena appearing less frequently. I have no real idea about what is going on but I can confirm that the video freezes due to some ticks not happening anymore somewhere.

@Substring, this is a good use case for deeper investigation, what's your opinion?

[Substring]

I have an up-to-date attract arch package (yeah, I read it's not related to attract). One could install it and see if the problem is solved, as well as update ffmpeg and eventually kernel. I may also suggest to give a try to the groovyarcade version I'm working on which is much more uptodate than the official one (almost cutting edge kernel available, updated almost everyday if any groovy specific package is updated). Still very beta though

Direct link for attract mode : https://github.com/substring/packages/releases/download/stable/attract-2.5.1-1-x86_64.pkg.tar.xz

Dunno if oomek or any other attract contributer has investigated this issue

[Banane]

thanks for keeping this up . I must say on my tests itīs not related to ffmpeg, thereīs no process of it in the back. It mostly even happens when I browse through the attract config menus.

first of all I'll test the new attract mode package. although I use arch on a daily basis and I could update the kernel... donīt I lose the 15Khz  output functionality? How does this work ?

@substring: do you have an iso-file of the version youīre working on ? It doesnīt have to be stable at all, als long I could test the attract menu. I'm really looking forward to your (final) version. I'd love to have arch's bleeding edge  background on my arcade machine as well.

this issue doesnīt have to be fixed tomorrow, as long as I donīt have to go back to stupid windows 
 

[Substring]

Yup I do ! Monthly releases for the iso : https://github.com/substring/os/releases

[Doozer]

I use fully up to date Arch (not GA based) and latest 5.3.1 kernel, the video playback issue is still here.

It is not related to the radeon driver, vesafb shows the same behavior.

I did some experiment to trace where the system get stuck. Unfortunately, I have not discovered the root cause of this.

In AM, freezes are always happening during video playback. Reason why I did separate tests with ffmpeg.

I am open to suggestions.

[Substring]

Rewrite the video backend to use GStreamer ?

...


*Running far away*

Except going the strace/gdb way, i really know nothing about ffmpeg or all the libav stuff.

[Banane]

@Doozer: Is it your whole system which gets stuck, or the frontend ? I still kill the process via ssh or from another runlevel, itīs just attract.
it does not freeze when you donīt have any videos in your snap folders ?

newbie tests I made after your replies:

• use of 2018 GA Version - same problem
• update of attract in 2019 GA version - same problem
• system updates or Substring's images: 15Khz/video driver/something trouble Iīm not into yet (is there a tutorial somehow? Iīd also like to test from an up to date version)

p.s. iīve to focus on distinguishing between mode with videos and mode without

[Substring]

I'm interested in feedback, it uses the gasetup from the good old groovyarcade (with a few hacks to have it working with my kernel). Otherwise, there is nothing much different

[Doozer]

@Doozer: Is it your whole system which gets stuck, or the frontend ? I still kill the process via ssh or from another runlevel, itīs just attract.
it does not freeze when you donīt have any videos in your snap folders ?

Thanks for testing and contributing.

Only the process displaying video stream is affected. My tests show that there is no relationship to the resolution used.
Playback starts fine after relaunching, it means no state corruption.

My assumptions would be more in favor of a library or backend issue.

[Banane]

Just for my personal understanding, ffmpeg is always involved in attract mode. Am I right ?

Now I was able to test with Substrings newest version as well ( @Substring: the fixes for the problem with the nvme SSD and the 15khz output after the first restart I sent you via PM, this is not related to this topic here   )

=> newer kernel , newer attract mode => no own games or snaps inserted => attract mode freezes after a while.
So if the answer to my question is yes, yep, than it could be an ffmpeg problem 

[Doozer]

Video playback inside AM is build upon ffmeg library.

Standalone ffmpeg player exhibits the same 'freezing' behavior.

The issue is limited to the process being unresponsive and a simple kill is sufficient to get rid of it. The OS layer is not impacted and state is nominal. Something is getting stuck and not ticking anymore somewhere in ffmpeg code. Long time ago I did some testing with other media player,  and only ffmpeg was showing this oddity.

[Substring]

@Doozer is the problem only happening in arch linux ? Or did you have the chance to test on other linux distros ?

[Doozer]

@Doozer is the problem only happening in arch linux ? Or did you have the chance to test on other linux distros ?

I must admit that I was too lazy to boot into another distro to check.

[Substring]

if you have a way to reproduce it, I can test on ubuntu + intel or nvidia GPU

[Banane]

Video playback inside AM is build upon ffmeg library.

Okay, I do understand this . But still, if you donīt  play any videos (like game snaps), just the pure menu, then ffmpeg is not used, right? (sorry, could be the language barrier).
My attract mode gets stuck EVEN when there is NO video playing .

Thatīs why I asked: Is attract mode always "combined" to ffmpeg? Is the pacman movement animation also done by ffmpeg?
If not, then there is still a 2nd problem going on.

[Doozer]

Video playback inside AM is build upon ffmeg library.

Okay, I do understand this . But still, if you donīt  play any videos (like game snaps), just the pure menu, then ffmpeg is not used, right? (sorry, could be the language barrier).
My attract mode gets stuck EVEN when there is NO video playing .

Thatīs why I asked: Is attract mode always "combined" to ffmpeg? Is the pacman movement animation also done by ffmpeg?
If not, then there is still a 2nd problem going on.

Indeed, if you experience freezes outside of video playback (screensaver, attract movie, ...) it is related to something else.

Have you already tried to trace the AM execution to see if a call or event can be identified?

[Doozer]

if you have a way to reproduce it, I can test on ubuntu + intel or nvidia GPU

Could you point me to the latest GA build considered stable? I will install it and check if I can produced it. I will also do a test with ubuntu on one machine.

At the moment the easiest way I found to reproduce it is to launch a video playback with ffplay and wait until the counter get stuck. The video freezes with the sound still playing few tens of second.

One point to mention, all my Radeon cards have their bios patched with Calamity's atom tool.

[Banane]

Have you already tried to trace the AM execution to see if a call or event can be identified?

This how this thread started  . Now weīre going for maybe two issues in the same one (a) freezes without ffmpeg, b) freezes due to ffmpeg)

For more than one hour now Iīm trying to find any further information. No dmesg, no "attract --loglevel debug", no "journalctl -f" ... any recommendations how to "trace" oder "debug" this ? 

I updated anything so far but the kernel, even reinstalled the sfml library

[Substring]

if you have a way to reproduce it, I can test on ubuntu + intel or nvidia GPU

Could you point me to the latest GA build considered stable? I will install it and check if I can produced it. I will also do a test with ubuntu on one machine.

Offical GA repo : https://drive.google.com/drive/folders/0B0NB2HYUHHktUFZXTWJfbHpzUlE

[Doozer]

For more than one hour now Iīm trying to find any further information. No dmesg, no "attract --loglevel debug", no "journalctl -f" ... any recommendations how to "trace" oder "debug" this ? 

Hi Banane,

I am sorry, I did not explain well what I meant by trace. I was thinking of the 'strace' command. This will show you the system calls and might give some clue on what is happening before the freeze occurs. Another option is to use the 'debug' command if the symbols are present in the binary. Otherwise AM needs to be recompiled for debug. Good how-to exist on the Net explaining how to attach, break and step during the execution.

Tonight I will do more tests wrt the video playback freezes. I think I have an idea about what is participating to the issue.

@Substring Thank you, I will do tests with your distro as well.

[Banane]

I was thinking of the 'strace' command.

Oh , sorry,  . I wasnīt thinking of this. Thanks, good idea

I attached the logfile of the strace right before it locks up. While in the idle main screen (no keyboard inputs made) itīs somehow polling something all the time but not receiving itīs wanted answer.

Could you image what this could be ?

18:50:00 [00007f572f46595d] recvmsg(4, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1\0\36\242\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 40
18:50:00 [00007f572f46595d] recvmsg(4, {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)

Beside that there is nothing special but the following.

What is this ? Could the lock up be related to some radeon driver issues ? (FYI - Iīm also using a HD6450 patched with a atom 15 bios, but the Ryzon 2400G also has a GPU integrated which is not attached to some monitor)

18:50:00 [00007f572cf1f0bb] getpid()    = 1064
18:50:00 [00007f572cf1f0bb] getpid()    = 1064
18:50:00 [00007f572cf4821b] ioctl(7, DRM_IOCTL_RADEON_GEM_BUSY, 0x7ffeb3ec3fc0) = 0
18:50:00 [00007f572f4619fd] futex(0x55e1dd26772c, FUTEX_WAKE_PRIVATE, 1) = 1
18:50:00 [00007f572f46485a] futex(0x55e1dd2676d8, FUTEX_WAKE_PRIVATE, 1) = 1
18:50:00 [00007f572cf4be3d] futex(0x55e1dda1f9f0, FUTEX_WAIT_BITSET, 2, NULL, FUTEX_BITSET_MATCH_ANY) = 0
18:50:00 [00007f572cf4821b] ioctl(7, DRM_IOCTL_RADEON_GEM_WAIT_IDLE, 0x7ffeb3ec41a0) = 0
18:50:00 [00007f572cf4821b] ioctl(7, DRM_IOCTL_RADEON_GEM_WAIT_IDLE, 0x7ffeb3ec41d0) = 0

Thanks 


p.s. okay, for the first try it seems to be related to the video card. I mean I had the same problem on my other mainboard/cpu and now I have it on the new hardware too (just the same video card).
I just ran it for half an hour with the internal GPU in vga mode and had no freezes so far.

I donīt really understand the other threads, if the internal vega APU is supported or not.

For testing purposes, can I get it to output 15 Khz ? or is there a decent card model you recommend ?

[Substring]

For the APU, check Calamity's A300 deskmini topic, he gave a few hints there.

[Doozer]

@substring, @banane

I have still some tests to performed to come to a conclusion. Take the following description as assumptions until fully confirmed.

I have tested the following cards so far, on a combination of different machines:
Radeon HD 5450
Radeon HD 2400 XT
Radeon HD 3650
Radeon HD 3450
Radeon HD 7450
Radeon HD 8990
Radeon HD 6450

I discovered something which is indeed related to the video portion. It is present in the stock kernel but related to 15kHz video modes. As said before, I need to do some more test to narrow down the issue. It could be the radeon driver, the 15kHz modelines, some acceleration functions (X)...

@banane, the strace output confirms that your freeze and the playback freeze are IDENTICAL issue.

@all: Can someone confirms that the freeze also happen when ATI card is used without ATOM BIOS patch?

@all: I was only able to spot the issue on ATI HW, if someone has issue with NVIDIA, INTEL, please provide your inputs. thank you.

What is known so far:
- not related to 15kHz patch
- not related to SDL2
- not related to ffmpeg (yes, I was wrong)
- issue can be triggered with EDID on stock kernel
- quite sure that ATOM PATCH is not the cause of this, I have issues on stock bios as well (more tests needed here)
- issue is not present when using the vesa driver

[Calamity]

Does this happen with all modes? Or is it specific to interlaced or progressive modes?

[Doozer]

Does this happen with all modes? Or is it specific to interlaced or progressive modes?

This is exactly what I have to confirm. All tests have been performed using interlaced resolutions up to now. 640x400i, 720x576i.... I was using the default resolution provided by EDID / 15kHz and suddenly I have spotted no issue when booting 31kHz when I hooked a standard LCD screen. I had to stop at that time but I will resume tests before coming to a conclusion.

[Doozer]

I can confirm that the issue is caused by X being used with an interlaced resolution.

Using progressive resolutions does not hang AM and video keep playing without freezes.

@Banane, this might also solve your issue. Let me know.

Now, it can be either Xorg, Radeon driver or DRM.

@All, can someone test NVIDIA with GA and interlaced resolution? If no issue here, it is definitely Radeon driver related

[Doozer]

The following progressive modelines are large enough for the frontend and display fine on CRT.

Code: [Select]

        modeline "640x240" 13.22 640 672 736 832 240 243 246 265 -hsync -vsync
        modeline "640x288" 13.10 640 672 736 832 288 289 292 309 -hsync -vsync


[Substring]

I'll try to test on Nvidia with GA (though I have absolutely no snap, will test on xterm + ffplay)

I had hoped amdgpu could work, but probably not with HD5xxx or HD6xxx

[Calamity]

I can confirm that the issue is caused by X being used with an interlaced resolution.

Using progressive resolutions does not hang AM and video keep playing without freezes.

@Banane, this might also solve your issue. Let me know.

Now, it can be either Xorg, Radeon driver or DRM.

@All, can someone test NVIDIA with GA and interlaced resolution? If no issue here, it is definitely Radeon driver related

Nice finding.

From Banane's log it looks like it hangs right after DRM_RADEON_GEM_WAIT_IDLE.

That call points here: https://elixir.bootlin.com/linux/v4.20/source/drivers/gpu/drm/radeon/radeon_gem.c#L476

Code: [Select]

ret = reservation_object_wait_timeout_rcu(robj->tbo.resv, true, true, 30 * HZ);
This is a random idea but the fact that interlaced modes are usually below 30 Hz might have something to do.

[Doozer]

Hi Calamity,

I have already patched this, but it is not the only place where reservation_object_wait_timeout_rcu is used. Nouveau and amdgpu should also be affected.

ATM, issue is still present with 10Hz 

[Calamity]

What if you just ret and ignore the wait?

EDIT: forget that, that'll probably break the whole driver.

[Banane]

Uuhh, I wasnīt here for half a day and now this movement  .Once again, thank you very much for digging into this, especially Doozer.

one by one:

Substring:
For the APU, check Calamity's A300 deskmini topic, he gave a few hints there.

Thanks, I did. For others maybe following the topic here, itīs this thread: http://forum.arcadecontrols.com/index.php/topic,159988.0.html.
I came to the conclusion that I will take care about the new Ryzen APUs later on. Right now there are other problems to solve. Instead I bought new used hardware for testing (1x Radeon R5 230 + 1x Radeon HD6870),  unfortunately no Intel or Nvidia.

Doozer:
@banane, the strace output confirms that your freeze and the playback freeze are IDENTICAL issue.
@all: Can someone confirms that the freeze also happen when ATI card is used without ATOM BIOS patch?

Great, makes it easier to follow and debug both problems if itīs actually just one.
I can confirm that the freezes also happen without the ATOM BIOS.  I both reflashed my HD6450 and tested the R5 230.

Doozer:
Using progressive resolutions does not hang AM and video keep playing without freezes.
@Banane, this might also solve your issue. Let me know.

How do I force AM to only output in progressive resolution ? edit: Okay, by a string search I found out itīs in /etc/X11/xorg.conf
Current modeLine: ModeLine          "648x480x60.00" 13.129200 648 672 736 840 480 482 488 521 -HSync -VSync interlace

I put in your recommended modelines for testing if AM freezes.  Sorry, Iīm not into all that stuff like  "resolution", "modelines" etc. I need to read and understand first. 640x480 progressive is not a 15Khz resolution anymore, right ? Thatīs why you did e.g 640x240  ?
Donīt know if this is a useful information, but i Have a dualSync Weiya Chassis (15Khz/24Khz). edit: 640x480 is VGA, 31Khz... okay, okay ... Can attract mode set to a low res fullscreen resolution or would it look  like crap ?
 
Also did you find a way to force the crash? Sometimes during my tests AM runs fine for more than about 10 minutes.
I know I could force it with video playback, as you said, but did you maybe figure out another way just for the menu issue ? 

[Doozer]

648x480 and 640x240 are 15kHz resolution. Progressive is the opposite of interlaced. 648x480 is an interlaced resolution, 2 fields (2 times half the picture organized in odd and even lines) to draw a full picture. 640x240 is a progressive resolution which is draw entirely in one pass.

If you check the 640x480 modeline definition, you will see the keyword "interlace" at the end of the line. Replace this line with a progressive mode like 640x480 and you issue with the freezes will be gone.

t is a temporary workaround until a proper solution is found. The drawback is half of the vertical resolution for the moment.

[Banane]

Thanks for explanation and forget about what I wrote - I now understand by myself AM needs a higher resolution to display all the information. If itīs in low res, than it's  640x480 interlaced - makes sense.

A even better workaround would be a trisync monitor 

I can confirm the actual problem is found. I have  it running in progressive mode for more than one hour now without any freezes 

[tym]

@All, can someone test NVIDIA with GA and interlaced resolution? If no issue here, it is definitely Radeon driver related

I haven’t tried with GA (yet) but RetroArch hasn’t frozen running an interlaced MAME game with an Nvidia card. It was having the same problem with a Radeon card.

[pocketpal]

So I guess this temporary workaround is still the best thing going?  I just got GroovyArcade going in my cabinet last night, and ran into this problem.