UDP stream interrupts + EMS reboot
Dear EvoStream-Team
I was experimenting with a Panasonic IP camera trying to get the live stream through UDP. UDP at the camera is supported and works if I access it directly (using VLC or any other player). Taking this one step further, I try to pull a live stream from the camera to the EMS using the following command (the camera has IP 192.168.1.10):
pullstream localstreamname=test uri=rtsp://192.168.1.10/MediaInput/h264/stream_1 forceTcp=0
This works as expected. The packet sniffer confirms that EMS pulls the stream over "RTP/AVP/UDP" (stated in RTSP handshake).
Then I try to access the stream using VLC using the following URL (192.168.1.19 is where EMS is installed):
rtmp://192.168.1.19/live/test
The stream can be accessed, but interrupts every few seconds. Then it seems to reconnect, plays for 5-10s and stops again. This is repeated over and over. I wanted to know the reason of the behavior, thus looked into the EMS logfile saying the following:
...
1440060209:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060209:6::0::Protocol [IJSONCLI(616)] unregistered from application: evostreamms
1440060224:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060224:3::0::Inbound connection accepted: (Far: 127.0.0.1:35902; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(617) <-> IHTT(618) <-> H4C(619) <-> [IJSONCLI(620)]
1440060224:3::0::command: version
1440060224:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060224:6::0::Protocol [IJSONCLI(620)] unregistered from application: evostreamms
1440060239:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060239:3::0::Inbound connection accepted: (Far: 127.0.0.1:35905; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(621) <-> IHTT(622) <-> H4C(623) <-> [IJSONCLI(624)]
1440060239:3::0::command: version
1440060239:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060239:6::0::Protocol [IJSONCLI(624)] unregistered from application: evostreamms
1440060254:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060254:3::0::Inbound connection accepted: (Far: 127.0.0.1:35908; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(625) <-> IHTT(626) <-> H4C(627) <-> [IJSONCLI(628)]
1440060254:3::0::command: version
1440060254:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060254:6::0::Protocol [IJSONCLI(628)] unregistered from application: evostreamms
1440060269:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060269:3::0::Inbound connection accepted: (Far: 127.0.0.1:35911; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(629) <-> IHTT(630) <-> H4C(631) <-> [IJSONCLI(632)]
1440060269:3::0::command: version
1440060269:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060270:6::0::Protocol [IJSONCLI(632)] unregistered from application: evostreamms
1440060285:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060285:3::0::Inbound connection accepted: (Far: 127.0.0.1:35914; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(633) <-> IHTT(634) <-> H4C(635) <-> [IJSONCLI(636)]
1440060285:3::0::command: version
1440060285:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060285:6::0::Protocol [IJSONCLI(636)] unregistered from application: evostreamms
1440060300:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060300:3::0::Inbound connection accepted: (Far: 127.0.0.1:35917; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(637) <-> IHTT(638) <-> H4C(639) <-> [IJSONCLI(640)]
1440060300:3::0::command: version
1440060300:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060300:6::0::Protocol [IJSONCLI(640)] unregistered from application: evostreamms
1440060315:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060315:3::0::Inbound connection accepted: (Far: 127.0.0.1:35920; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(641) <-> IHTT(642) <-> H4C(643) <-> [IJSONCLI(644)]
1440060315:3::0::command: version
1440060315:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060315:6::0::Protocol [IJSONCLI(644)] unregistered from application: evostreamms
1440060330:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060330:3::0::Inbound connection accepted: (Far: 127.0.0.1:35923; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(645) <-> IHTT(646) <-> H4C(647) <-> [IJSONCLI(648)]
1440060330:3::0::command: version
1440060330:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060330:6::0::Protocol [IJSONCLI(648)] unregistered from application: evostreamms
1440060345:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060345:3::0::Inbound connection accepted: (Far: 127.0.0.1:35926; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(649) <-> IHTT(650) <-> H4C(651) <-> [IJSONCLI(652)]
1440060345:3::0::command: version
1440060345:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060345:6::0::Protocol [IJSONCLI(652)] unregistered from application: evostreamms
1440060360:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060360:3::0::Inbound connection accepted: (Far: 127.0.0.1:35929; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(653) <-> IHTT(654) <-> H4C(655) <-> [IJSONCLI(656)]
1440060360:3::0::command: version
...
The ‘unregistered’ messages are repeated in the same interval as the stream in the VLC is interrupted. But this is not the only issue I faced. As soon as I hit the stop button at the VLC player, the logfile says the following:
...
1440060360:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060360:6::0::Protocol [IJSONCLI(656)] unregistered from application: evostreamms
1440060375:4::0::Handlers count changed: 19->20 IOHT_TCP_CARRIER
1440060375:3::0::Inbound connection accepted: (Far: 127.0.0.1:35932; Near: 127.0.0.1:7777) CTCP(18) <-> TCP(657) <-> IHTT(658) <-> H4C(659) <-> [IJSONCLI(660)]
1440060375:3::0::command: version
1440060375:4::0::Handlers count changed: 20->19 IOHT_TCP_CARRIER
1440060375:6::0::Protocol [IJSONCLI(660)] unregistered from application: evostreamms
1440060381:0::0::Unable to read data from connection: (Far: 127.0.0.1:53958; Near: 127.0.0.1:1113) CTCP(16) <-> [TCP(18)] <-> BVAR(19). Error was (104): Connection reset by peer
1440060381:4::0::Handlers count changed: 19->18 IOHT_TCP_CARRIER
1440060381:6::0::Protocol [BVAR(19)] unregistered from application: evostreamms
1440060381:0::0::Unable to read data from connection: (Far: 127.0.0.1:43934; Near: 127.0.0.1:1936) CTCP(17) <-> [TCP(603)] <-> IR(604). Error was (104): Connection reset by peer
1440060381:4::0::Handlers count changed: 18->17 IOHT_TCP_CARRIER
1440060381:3::0::Stream ONR(7) with name `test` from protocol IR(604) unregistered from origin application `evostreamms`
1440060381:6::0::Protocol [IR(604)] unregistered from application: evostreamms
1440060382:4::0::Handlers count changed: 17->18 IOHT_TCP_CONNECTOR
1440060382:2::0::Unknown pid 4123 terminated
1440060382:4::0::***CONNECT ERROR: Unable to connect to: 127.0.0.1:37682
1440060382:2::0::Edge with pid 4123 went down
1440060382:4::0::Handlers count changed: 18->17 IOHT_TCP_CONNECTOR
1440060390:4::0::Handlers count changed: 17->18 IOHT_TCP_CARRIER
1440060390:3::0::Inbound connection accepted: (Far: 127.0.0.1:35936; Near: 127.0.0.1:7777) CTCP(16) <-> TCP(661) <-> IHTT(662) <-> H4C(663) <-> [IJSONCLI(664)]
1440060390:3::0::command: version
1440060390:4::0::Handlers count changed: 18->17 IOHT_TCP_CARRIER
1440060390:6::0::Protocol [IJSONCLI(664)] unregistered from application: evostreamms
1440060405:4::0::Handlers count changed: 17->18 IOHT_TCP_CARRIER
1440060405:3::0::Inbound connection accepted: (Far: 127.0.0.1:35939; Near: 127.0.0.1:7777) CTCP(16) <-> TCP(665) <-> IHTT(666) <-> H4C(667) <-> [IJSONCLI(668)]
1440060405:3::0::command: version
1440060405:4::0::Handlers count changed: 18->17 IOHT_TCP_CARRIER
1440060405:6::0::Protocol [IJSONCLI(668)] unregistered from application: evostreamms
1440060420:4::0::Handlers count changed: 17->18 IOHT_TCP_CARRIER
1440060420:3::0::Inbound connection accepted: (Far: 127.0.0.1:35942; Near: 127.0.0.1:7777) CTCP(16) <-> TCP(669) <-> IHTT(670) <-> H4C(671) <-> [IJSONCLI(672)]
1440060420:3::0::command: version
1440060420:4::0::Handlers count changed: 18->17 IOHT_TCP_CARRIER
1440060420:6::0::Protocol [IJSONCLI(672)] unregistered from application: evostreamms
1440060422:2::0::Gracefully shutdown
1440060422:0::0::Unable to execute epoll_wait: (9) Bad file descriptor
1440060422:2::0::Shutting down protocols manager
1440060422:6::0::Protocol T(2) <-> [TMR(2)] unregistered from application: evostreamms
1440060422:6::0::Protocol T(3) <-> [TMR(3)] unregistered from application: evostreamms
1440060422:6::0::Protocol T(4) <-> [TMR(4)] unregistered from application: evostreamms
1440060422:6::0::Protocol T(5) <-> [TMR(5)] unregistered from application: evostreamms
1440060422:3::0::Stream INP(6) with name `test` from protocol RTSP(561) unregistered from origin application `evostreamms`
1440060422:6::0::Protocol (Far: 192.168.1.10:554; Near: 192.168.1.19:50700) CTCP(14) <-> TCP(560) <-> [RTSP(561)] unregistered from application: evostreamms
1440060422:0::0::Unable to get the timer protocol
1440060422:6::0::Protocol (Bound on: 0.0.0.0:48716) CUDP(15) <-> UDP(562) <-> [IRTP(563)] unregistered from application: evostreamms
1440060422:6::0::Protocol (Bound on: 0.0.0.0:48717) CUDP(13) <-> UDP(564) <-> [RTCP(565)] unregistered from application: evostreamms
1440060422:4::0::Handlers count changed: 17->16 IOHT_TIMER
1440060422:4::0::Handlers count changed: 16->15 IOHT_TIMER
1440060422:4::0::Handlers count changed: 15->14 IOHT_TIMER
1440060422:4::0::Handlers count changed: 14->13 IOHT_TIMER
1440060422:4::0::Handlers count changed: 13->12 IOHT_TIMER
1440060422:4::0::Handlers count changed: 12->11 IOHT_TCP_CARRIER
1440060422:4::0::Handlers count changed: 11->10 IOHT_UDP_CARRIER
1440060422:4::0::Handlers count changed: 10->9 IOHT_UDP_CARRIER
1440060422:4::0::Handlers count changed: 9->8 IOHT_TIMER
1440060422:2::0::Shutting down I/O handlers manager
1440060422:4::0::Handlers count changed: 8->7 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 7->6 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 6->5 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 5->4 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 4->3 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 3->2 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 2->1 IOHT_ACCEPTOR
1440060422:4::0::Handlers count changed: 1->0 IOHT_ACCEPTOR
1440060422:2::0::Unregister and delete default protocol handler
1440060422:2::0::Shutting down applications
1440060422:2::0::Delete the configuration
1440060422:2::0::Doing final OpenSSL cleanup
1440060422:2::0::Shutting down the logger leaving you in the dark. Bye bye... :(
As visible, EMS automatically shuts down. Another instance is launched afterwards. To be honest, I don’t know whether it’s lauched because of the minutely cron job installed on the system to check and start evostream, or automatically by the crashing evostream instance. Nevertheless, the crash should not happen.
Here are some further tests I did:
- omitting the forceTcp parameter or setting it to 1 does not resolve the stream interrupts. Thus this must be an incompatibility of EMS with Panasonic’s UDP mode. The ‘unregistered’ messages are still repeating.
- omitting the forceTCP parameter or setting it to 1 does not prevent the EMS from shutting down.