Local evaluation

[PolyProgrammistParticipant]

Hello,
I am facing a problem with local evaluation.
The command ‘dts challenges evaluate’ fails.
It says it can’t build a connect to docker daemon.

I attach the logs
https://drive.google.com/file/d/1Nlx-CoukWoFkW8Vfg5QX6Mk2a40zFOT1/view?usp=sharing

Best regards,
Vadim Volodin

[Florian GolemoParticipant]

This might be a problem with your docker configuration. Could you send us the output of docker info, please?

[Andrea CensiKeymaster]

This is a known problem – we are trying to understand the reason.

At the bottom here you can find a workaround:

http://docs.duckietown.org/DT18/AIDO/out/cm_first.html

[PolyProgrammistParticipant]

Oh, I didn’t try the troubleshooting, thanks.
Here is docker info output https://drive.google.com/file/d/1XpVHTUgaOK8z1YsFfmrJlpPBxrMbKgkq/view?usp=sharing

[heyt0nyParticipant]

Hello!

I’m facing error when running dts challenges evaluate. I tried it on all repos with “template” on github and it raises:

`
ERROR:runner-local:Traceback (most recent call last):
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 35, in runner_local_main
runner_local_main_()
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 84, in runner_local_main_
client = check_docker_environment()
File “/project/src/duckietown-shell/lib/dt_shell/env_checks.py”, line 36, in check_docker_environment
check_user_in_group(‘docker’)
File “/project/src/duckietown-shell/lib/dt_shell/env_checks.py”, line 79, in check_user_in_group
active_groups = get_active_groups(username=None)
File “/project/src/duckietown-shell/lib/dt_shell/env_checks.py”, line 104, in get_active_groups
raise Exception(str(e))
Exception: Command ‘[‘groups’]’ returned non-zero exit status 1

INFO:dts:status: exited
ERROR:dts:The container exited.
INFO:dts:Logs saved at evaluator.log
`

evaluator.log is empty.

[PolyProgrammistParticipant]

Hello, my logs now are as following, but my username is in group docker

DEBUG:dts:checking docker environment
INFO:dts:The server URL is: https://challenges.duckietown.org/v3
INFO:dts:Updating container duckietown/dt-challenges-evaluator:v3
INFO:dts:Starting container local-evaluator with duckietown/dt-challenges-evaluator:v3
INFO:dts:Container command: dt-challenges-evaluate-local –output output
INFO:dts:status: running
INFO:dt-challenges-runner:dt-challenges-runner 3.1.41
INFO:duckietown-challenges:duckietown-challenges 3.2.3
INFO:dts:duckietown-shell 3.0.24
DEBUG:runner-local:Running in directory /home/polyprogrammist/Documents/duckietown/agent/lf1first
DEBUG:dts:Checking docker environment
ERROR:runner-local:Traceback (most recent call last):
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 35, in runner_local_main
runner_local_main_()
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 84, in runner_local_main_
client = check_docker_environment()
File “/project/src/duckietown-shell/lib/dt_shell/env_checks.py”, line 36, in check_docker_environment
check_user_in_group(‘docker’)
File “/project/src/duckietown-shell/lib/dt_shell/env_checks.py”, line 86, in check_user_in_group
raise InvalidEnvironment(msg)
InvalidEnvironment: The user is not in group “docker”.

It belongs to groups: root.

Note that when you add a user to a group, you need to login in and out.

[PolyProgrammistParticipant]

I did the troubleshooting. It helped. But for the second attempt it broke.
Step 10/10 : CMD python solution.py
—> Using cache
—> dd5a79cf7b39
Successfully built dd5a79cf7b39
Successfully tagged myimage:latest
INFO:runner-local:steps: [u’step1-simulation’, u’step2-scoring’]
INFO:runner-local:Now considering step “step1-simulation”
ERROR:runner-local:Traceback (most recent call last):
File “/home/polyprogrammist/.local/lib/python2.7/site-packages/duckietown_challenges_runner/runner_local.py”, line 35, in runner_local_main
runner_local_main_()
File “/home/polyprogrammist/.local/lib/python2.7/site-packages/duckietown_challenges_runner/runner_local.py”, line 129, in runner_local_main_
shutil.rmtree(wd)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 266, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File “/usr/lib/python2.7/shutil.py”, line 264, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: ‘output-local-evaluation/step1-simulation.tmp/challenge-evaluation-output/episodes/ep000/log.bag’

[PolyProgrammistParticipant]

test

[PolyProgrammistParticipant]

From Anna Tsalapova. She couldn’t create a post here because of some issue of the forum
>

Hello
I’m facing a similar problem. When I first run dts challenges evaluate command (on the random template) it works fine. Then I change solution.py a bit and run dts challenges evaluate again and I failes with the problem

ERROR:runner-local:Traceback (most recent call last):
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 35, in runner_local_main
runner_local_main_()
File “/project/src/duckietown_challenges_runner/runner_local.py”, line 124, in runner_local_main_
shutil.rmtree(wd_final)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “/usr/lib/python2.7/shutil.py”, line 266, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File “/usr/lib/python2.7/shutil.py”, line 264, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: ‘output/step1-simulation/challenge-evaluation-output/episodes/ep002/log.bag’

Full logs you can find here: https://drive.google.com/file/d/1iMuoBbhU9BeJugatU9AdE9Hqkm17cBjS/view?usp=sharing

[Bhairav MehtaParticipant]

It seems like you may have an older version of duckietown-shell

If you are running locally, can you do
sudo rm -rf output && dts update

If you’re running in Docker, just run the second command.

Try this and see if it works?

[anna.tsalapovaParticipant]

I’m running locally and it did not help, I got the same error.

[Bhairav MehtaParticipant]

Could you explain what was changed in solution.py? Some lines inside are critical to correct scoring.

[anna.tsalapovaParticipant]

I just changed threshold value in our simple algorithm, just one number.

[Andrea CensiKeymaster]

@polyprogrammist – can you remove the directory “output” and try again?

I think it’s a left over from before when we wrote the wrong permissions.

Alternatively use dts challenges evaluate --output DIR where DIR does not exist yet.

[PolyProgrammistParticipant]

Hello, It works with removing an output. As soon as I understood, we recalculate a lot of data doing this way? If so, it’s easier to send submission to server than evaluate locally.

We changed the rights of the following directory and that worked: output/step1-simulation/challenge-evaluation-output/episodes
Could you make the permissions for it, please?

[Author]

Posts