Liquidsoap crashes using LastFM.submit

[GraveR]

When using the LastFM plugin. LS crashes after some hours with sig11.

Backtrace (doesn't seem too useful):

#0  0x00000008014f3dcc in pthread_testcancel () from /lib/libpthread.so.2
#1  0x00000008014e2cd8 in sigaction () from /lib/libpthread.so.2
#2  0x00000008014ec852 in pthread_mutexattr_init () from /lib/libpthread.so.2
#3  0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffffbff000

[GraveR]

The used script

[toots]

Hi GraveR !

Indeed, the trace is not really usefull..

First of all, from the script example, I guess you meant lastfm.submit and not input.lastfm.

Most of the code in lastfm.submit is plain ocaml processing on which I don't really believe this could happen. I would rather suspect something going wrong with ocamlnet, which is used to submit http requests, or, less likely, from xml-light which is used for xml parsing.

Howeverm I'll be gled to fix this, but I clearly needs more details on the issue...

[GraveR]

Yes, this is indeed lastfm.submit. What information do you need to further investigate?

[toots]

Well, the backtrace was a good idea, but it doesn't seem usefull.

I don't really know how to gather more informations. If I could reproduce the issue, I would add dummy Printf.printf calls to try to isolate the part of the code that triggers the crash..

For instance, I would look at duration computation in trunk/liquidsoap/src/tools/liqfm.ml and the http request function defined on top of ocamlnet, in the head of trunk/ocaml-lastfm/src/lastfm.ml.

You may also loop on some example taken from trunk/ocaml-lastfm/examples/submit.ml to try to reproduce the issue outside of liq..

Ha, also some verbose logs -- set("log.level",5) -- could help to find out the point where it crashes.

[GraveR]

The culprit indeed seems to have something to do with LastFM, this being the last output before the Segfault:

2008/07/31 21:06:45 [src_3139:4] queue length 1323000+=1323000 (rid 13)
2008/07/31 21:06:45 [liqfm:4] Submiting New Four -- Halverwege Amsterdam & Bremershaven to lastfm

Will (try to, since I know squat about ocaml) modify the sources and find the little bugger.

[mrpingouin]

Just a quick note to say that the backtrace from the first thread is never interesting, instead the backtraces of all threads have more chances to help. Here's how to get them: http://savonet.rastageeks.org/wiki/Debugging

[GraveR]

The box has been upgraded from FreeBSD to Ubuntu Linux, the problem is not relevant for me anymore.

[GraveR]

Same thing happens on Ubuntu Linux when LastFM.submit is used, reopening.

[toots]

Ok, I'll try to reproduce if this hapenned in ubuntu too.

Could you give me the version of ocamlnet that you use please ?

[GraveR]

The stock Ubuntu 8.04 one: libocamlnet-ocaml 2.2.8.1-1build2 (amd64)

[GraveR]

Hmz. It has been running stable for over three consecutive days now. Maybe it somehow helped when I recompiled it with debugging enabled?

We'll see if it still works in a week or so. I'll keep you posted.

[GraveR]

False alarm, my mistake. Sorry about the fuss about 'crashing' on Ubuntu. Seems it was caused by something else.

2008/09/13 02:02:39 [source:4] src_77009 gets down
2008/09/13 02:02:39 [single:2] Couldn't get an audio request! Make sure you did not override audio=false in request.create().
2008/09/13 02:02:39 [source:4] src_77041 gets up
2008/09/13 02:02:39 [threads:1] thread "generic queue #1" aborts with exception Unix.Unix_error(15, "pipe", "") !
2008/09/13 02:02:39 [source:4] src_77040 gets up
2008/09/13 02:02:39 [main:3] Shutdown started!
2008/09/13 02:02:39 [threads:3] thread "non-blocking queue #1" exited (1 remaining)
2008/09/13 02:02:39 [root:2] Root shutdown
2008/09/13 02:02:39 [source:4] debug(dot)mp3 gets down
2008/09/13 02:02:39 [source:4] src_3491 gets down
2008/09/13 02:02:39 [source:4] src_3474 gets down
2008/09/13 02:02:39 [threads:3] thread "root" exited (0 remaining)
2008/09/13 02:02:42 [main:3] Cleaning downloaded files...
2008/09/13 02:02:42 >>> LOG END

Ofcourse, audio=false WAS overridden by audio=true in the script

[toots]

Humm

It is possible that sometimes lastfm.submit fails because the metadata does not come from a real file for instance.

However, this shouldn't crash the whole application. I'll look for an uncatched exception there..

[GraveR]

And we have a winner ;)

Liquidsoap 0.3.8.1.

Thread 6 (process 1522):
#0  0x00007f7f26e734f9 in do_sigwait () from /lib/libpthread.so.0
#1  0x00007f7f26e7359d in sigwait () from /lib/libpthread.so.0
#2  0x00000000005d2206 in caml_wait_signal ()
#3  0x00000000005e7af4 in caml_c_call ()
#4  0x00007fff2f749720 in ?? ()
#5  0x00000000004bba29 in camlDtools__wait_signal_340 ()
#6  0x00007fff2f749720 in ?? ()
#7  0x00000000004bbbe0 in camlDtools__main_343 ()
#8  0x00007f7f27727bd8 in ?? ()
#9  0x00007f7f2770aff8 in ?? ()
#10 0x00007f7f1ece6518 in ?? ()
#11 0x00000000004bbd4c in camlDtools__catch_353 ()
#12 0x00007fff2f749780 in ?? ()
#13 0x00000000004bbc75 in camlDtools__catch_353 ()
#14 0x00007f7f1ece64e8 in ?? ()
#15 0x00007f7f27727bd8 in ?? ()
#16 0x0000000000893cb8 in camlDtools__59 ()
#17 0x0000000000414a53 in camlMain__entry ()
#18 0x00007f7f27727bd8 in ?? ()
#19 0x00007f7f27727bb8 in ?? ()
#20 0x00000000008089b0 in camlLang_lastfm ()
#21 0x000000000040dc7f in caml_startup__code_begin ()
#22 0x00000000004a4bf9 in camlRqueue__insert_pred_98 ()
#23 0x00000000005e7b4a in caml_start_program ()
#24 0x0000000000000000 in ?? ()

Thread 5 (process 1523):
#0  0x00007f7f23f3cda2 in select () from /lib/libc.so.6
#1  0x00000000005d3127 in caml_thread_tick ()
#2  0x00007f7f26e6b3f7 in start_thread () from /lib/libpthread.so.0
#3  0x00007f7f23f43b2d in clone () from /lib/libc.so.6
#4  0x0000000000000000 in ?? ()

Thread 4 (process 1524):
#0  0x00007f7f23f3cda2 in select () from /lib/libc.so.6
#1  0x00000000005d5245 in unix_select ()
#2  0x00000000005e7af4 in caml_c_call ()
#3  0x000000000000000d in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007f7f277231a8 in ?? ()
#7  0x0000000000892978 in camlDuppy__25 ()
#8  0x0000000000882e10 in camlTutils__39 ()
#9  0x00007f7f27723510 in ?? ()
#10 0x00007f7f1ecf5bf8 in ?? ()
#11 0x00007f7f1ecf5bf8 in ?? ()
#12 0x00000000004b72ac in camlDuppy__queue_199 ()
#13 0x00007f7f1f1d24b0 in ?? ()
#14 0x00007f7f1e4ab1a0 in ?? ()
#15 0x0000000000882de0 in camlTutils__36 ()
#16 0x0000000000882e10 in camlTutils__39 ()
#17 0x00007f7f1ecf5bf8 in ?? ()
#18 0x00000000004a1822 in camlTutils__fun_350 ()
#19 0x0000000040fb0fc0 in ?? ()
#20 0x00000000004a169e in camlTutils__fun_350 ()
#21 0x00007f7f26e6ede9 in _L_unlock_163 () from /lib/libpthread.so.0
#22 0x00007f7f26e6eb26 in __pthread_mutex_unlock_usercnt ()
   from /lib/libpthread.so.0
#23 0x0000000040fb1950 in ?? ()
#24 0x0000000000000000 in ?? ()

Thread 3 (process 1525):
#0  0x00007f7f26e6fb99 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/libpthread.so.0
#1  0x00000000005d2430 in caml_condition_wait ()
#2  0x00000000005e7af4 in caml_c_call ()
#3  0x00007f7f277218c0 in ?? ()
#4  0x00007f7f1e4ab170 in ?? ()
#5  0x0000000000892730 in camlDuppy__12 ()
#6  0x0000000000882e10 in camlTutils__39 ()
#7  0x00007f7f1ecf5bf8 in ?? ()
#8  0x00000000004a1822 in camlTutils__fun_350 ()
#9  0x00000000421d1fc0 in ?? ()
#10 0x00000000004a169e in camlTutils__fun_350 ()
#11 0x00007f7f26e6ede9 in _L_unlock_163 () from /lib/libpthread.so.0
#12 0x00007f7f26e6eb26 in __pthread_mutex_unlock_usercnt ()
   from /lib/libpthread.so.0
#13 0x00000000421d2950 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 2 (process 1526):
#0  0x00007f7f26e6fb99 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/libpthread.so.0
#1  0x00000000005d2430 in caml_condition_wait ()
#2  0x00000000005e7af4 in caml_c_call ()
#3  0x00007f7f1e4ab128 in ?? ()
#4  0x00000000004b9449 in camlDtools__thread_348 ()
#5  0x00000000429d2fc0 in ?? ()
#6  0x00000000004b93b3 in camlDtools__thread_348 ()
#7  0x00007f7f26e759a8 in __PRETTY_FUNCTION__.7137 () from /lib/libpthread.so.0
#8  0x00007f7f1e4aaf30 in ?? ()
#9  0x00007f7f26e6ede9 in _L_unlock_163 () from /lib/libpthread.so.0
#10 0x00007f7f26e6eb26 in __pthread_mutex_unlock_usercnt ()
   from /lib/libpthread.so.0
#11 0x00000000429d3950 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 1 (process 1527):
#0  0x00000000005d9fd8 in mark_slice ()
#1  0x00000000005da909 in caml_major_collection_slice ()
#2  0x00000000005daece in caml_minor_collection ()
#3  0x00000000005daf31 in caml_check_urgent_gc ()
#4  0x00000000005dbb8f in caml_alloc ()
#5  0x00000000005ca975 in ocaml_mad_decode_frame_float (madf=140184000904336)
    at mad_stubs.c:443
#6  0x00000000005e7af4 in caml_c_call ()
#7  0x00000000431d3f90 in ?? ()
#8  0x000000000048159a in camlMp3__fill_149 ()
#9  0x00007f7f1e4e2af8 in ?? ()
#10 0x00007f7f2772c528 in ?? ()
#11 0x00007f7f21a0bdb0 in ?? ()
#12 0x000000000047a894 in camlRequest_source__fun_463 ()
#13 0x00007f7f21a0be58 in ?? ()
#14 0x000000000047af73 in camlRequest_source__method_get_frame_359 ()
#15 0x00007f7f2772cdb0 in ?? ()
#16 0x00007f7f1e4e2af8 in ?? ()
#17 0x00007f7f1ecd94d8 in ?? ()
#18 0x00007f7f1e52cbd8 in ?? ()
#19 0x4d69f4f84e10626a in ?? ()
#20 0x00000000004970e5 in camlSource__method_get_367 ()
#21 0x0000000000000001 in ?? ()
#22 0x000000000049e65d in camlFmt__samples_of_ticks_111 ()
#23 0x00007f7f1e4e2af8 in ?? ()
#24 0x00007f7f1ecd9668 in ?? ()
#25 0x00007f7f1e52cbd8 in ?? ()
#26 0x000000000045b81d in camlNormalize__method_get_frame_164 ()
#27 0x0000000000000001 in ?? ()
#28 0x00007f7f1e4e2af8 in ?? ()
#29 0x520440bd52d06282 in ?? ()
#30 0x5197713551196b09 in ?? ()
#31 0x507069865104193b in ?? ()
#32 0x50aacf4a50346d2c in ?? ()
#33 0x4fd79c124f95cc3c in ?? ()
#34 0x4b118c174ffa833e in ?? ()
#35 0x4ab7c88a4ac0c5b4 in ?? ()
#36 0x500b60f1515c3b21 in ?? ()
#37 0x4dda37d04c713785 in ?? ()
#38 0x4cabcfda4c56faa0 in ?? ()
#39 0x4f0964834e85b51e in ?? ()
#40 0x4f2e921d4f02dc2b in ?? ()
#41 0x502ab9294f630f7b in ?? ()
#42 0x527a7e8e52695584 in ?? ()
#43 0x00007f7f1ec9b430 in ?? ()
#44 0x00007f7f1e4e1d38 in ?? ()
#45 0x510f2e335198b9e9 in ?? ()
#46 0x4f3da57750822200 in ?? ()
#47 0x507af78251211efe in ?? ()
#48 0x00000000004970e5 in camlSource__method_get_367 ()
#49 0x0000000000000001 in ?? ()
#50 0x000000000049e65d in camlFmt__samples_of_ticks_111 ()
#51 0x00007f7f1e4e2af8 in ?? ()
#52 0x00007f7f1ec9b310 in ?? ()
#53 0x00007f7f1e4e1d38 in ?? ()
#54 0x000000000043a663 in camlCompress__method_get_frame_181 ()
#55 0x0000000000000001 in ?? ()
#56 0x00007f7f1e4e2af8 in ?? ()
#57 0x4f20ebdd5072f93b in ?? ()
#58 0x4f3ca3364ebca371 in ?? ()
#59 0x5188557851362355 in ?? ()
#60 0x00007f7f1ec50340 in ?? ()
#61 0x00007f7f1e4e1f90 in ?? ()
#62 0x4f31a0f44f3a5b25 in ?? ()
#63 0x4ea055a64f3c01b6 in ?? ()
#64 0x4f0af4394f024c3f in ?? ()
#65 0x4b8385924b185251 in ?? ()
#66 0x5436c6eb4bd23ac6 in ?? ()
#67 0x4e22cd81518f800e in ?? ()
#68 0x4eb14e3a4fc053c2 in ?? ()
#69 0x4fca9e904eb746bf in ?? ()
#70 0x5045f44650899247 in ?? ()
#71 0x503bd78250b1abe7 in ?? ()
#72 0x53b1eb0451436dee in ?? ()
#73 0x525e3be9541fccbf in ?? ()
#74 0x00000000004970e5 in camlSource__method_get_367 ()
#75 0x0000000000000001 in ?? ()
#76 0x519e361a5207366b in ?? ()
#77 0x00007f7f1e4e2af8 in ?? ()
#78 0x00007f7f1ec588e0 in ?? ()
#79 0x00007f7f1e4e1f90 in ?? ()
#80 0x0000000000462fa8 in camlOn_metadata__method_get_frame_140 ()
#81 0x00007f7f1e4e2af8 in ?? ()
#82 0x00007f7f1ecb12c8 in ?? ()
#83 0x00007f7f1e4e1f08 in ?? ()
#84 0x0000000000000001 in ?? ()
#85 0x51a858ba51a24b58 in ?? ()
#86 0x00000000004970e5 in camlSource__method_get_367 ()
#87 0x0000000000000001 in ?? ()
#88 0x00000000005e7b4a in caml_start_program ()
#89 0x00007f7f1e4e2af8 in ?? ()
#90 0x00007f7f1ecb1450 in ?? ()
#91 0x00007f7f1e4e1f08 in ?? ()
#92 0x000000000045f6e8 in camlSwitch__method_get_frame_544 ()
#93 0x00007f7f1e4e2af8 in ?? ()
#94 0x00007f7f1eca5ec0 in ?? ()
#95 0x00007f7f1e4e2288 in ?? ()
#96 0x00007f7f1e4e1f08 in ?? ()
#97 0x00007f7f1f87fad8 in ?? ()
#98 0x00007f7f1e4b3078 in ?? ()
#99 0x00000000429d2f10 in ?? ()
#100 0x00000000004970e5 in camlSource__method_get_367 ()
#101 0x0000000000000001 in ?? ()
#102 0x000000000049e65d in camlFmt__samples_of_ticks_111 ()
#103 0x00007f7f1e4e2af8 in ?? ()
#104 0x00007f7f1eca5d50 in ?? ()
#105 0x00007f7f1e4e2288 in ?? ()
#106 0x0000000000465450 in camlFade__method_get_frame_286 ()
#107 0x00007f7f1f84b8a0 in ?? ()
#108 0x00000000005dab3d in caml_oldify_one ()
#109 0x0000000000000001 in ?? ()
#110 0x00007f7f1e4e2af8 in ?? ()
#111 0x00007f7f1ecba128 in ?? ()
#112 0x00007f7f1e4b2e80 in ?? ()
#113 0x00000000004970e5 in camlSource__method_get_367 ()
#114 0x0000000000000001 in ?? ()
#115 0x000000000049e65d in camlFmt__samples_of_ticks_111 ()
#116 0x00007f7f1e4e2af8 in ?? ()
#117 0x00007f7f1ecba008 in ?? ()
#118 0x00007f7f1e4b2e80 in ?? ()
#119 0x00000000004659f0 in camlFade__method_get_frame_321 ()
#120 0x0000000000000000 in ?? ()

[mrpingouin]

Thanks. I don't have time to look more into it, but that trace sounds like a faulty C binding. All threads are politely waiting, and one thread triggered the GC, which causes a segfault.

[toots]

Hi !

Sorry, it seems we lagged a bit here.

Indeed, the issue looks like a binding bug. However, it is not necessarily ocaml-mad, which is mentioned here. As David explained, the binding tried to allocate a new variable for ocaml, which trigered the garbage collector in order to make some room, and then segfaulted when trying to perform the cleanup.

Hence, it is more likely that this is caused by a wrong allocation in *another* binding...

Did you experience this issue recently ? I am considering a move from ocamlnet to ocaml-curl, would you be able to test a patched lastfm not using ocamlnet ?

[wiki]

wrought iron stair railings decorative railing deck railings