trainer Fails When xcodebuild Quits With Exit Status 65

[SlaunchaMan]

Even though I have the fail_build setting set to false, occasionally if xcodebuild returns an exit status of 65, trainer will fail.

A snippet from the log

** TEST FAILED **
[09:13:18]: Exit status: 65
+--------------------+-----+
|       Test Results       |
+--------------------+-----+
| Number of tests    | 202 |
| Number of failures | 1   |
+--------------------+-----+

[09:13:18]: ---------------------
[09:13:18]: --- Step: trainer ---
[09:13:18]: ---------------------

+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                        Summary for trainer 0.8.0                                                                                        |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| output_directory | ./fastlane/test_output                                                                                                                                                               |
| extension        | .junit                                                                                                                                                                               |
| fail_build       | false                                                                                                                                                                                |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[09:13:18]: `Helper.gem_path` is deprecated. Use the `ROOT` constant from the appropriate tool module instead.
Successfully generated './fastlane/test_output/TestSummaries.junit'
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                     Lane Context                                                                                                      |
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| DEFAULT_PLATFORM           | ios                                                                                                                                                                                      |
| PLATFORM_NAME              | ios                                                                                                                                                                                      |
| LANE_NAME                  | ios pr                                                                                                                                                                                   |
| BUILD_NUMBER               | 81.289                                                                                                                                                                                   |
| SCAN_DERIVED_DATA_PATH     | /Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe                                                                                          |
| SCAN_GENERATED_PLIST_FILES | ["/Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe/Logs/Test/Run-REDACTED-2018.09.07_09-09-41--0400.xcresult/TestSummaries.plist"] |
| SCAN_GENERATED_PLIST_FILE  | /Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe/Logs/Test/Run-REDACTED-2018.09.07_09-09-41--0400.xcresult/TestSummaries.plist     |
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[09:13:18]: Unit tests failed

+------+-----------------------------------------+-------------+
|                       fastlane summary                       |
+------+-----------------------------------------+-------------+
| Step | Action                                  | Time (in s) |
+------+-----------------------------------------+-------------+
| 1    | default_platform                        | 0           |
| 2    | Verifying fastlane version              | 0           |
| 3    | default_platform                        | 0           |
| 4    | import_from_git                         | 0           |
| 5    | xcversion                               | 1           |
| 6    | cocoapods                               | 15          |
| 7    | Switch to ios incrementBuildNumber lane | 0           |
| 8    | increment_build_number                  | 2           |
| 9    | Switch to ios runTestsWithCoverage lane | 0           |
| 10   | reset_simulator_contents                | 20          |
| 11   | swiftlint                               | 1           |
| 12   | scan                                    | 237         |
| 💥   | trainer                                 | 0           |
+------+-----------------------------------------+-------------+

[09:13:18]: fastlane finished with errors

[!] Unit tests failed

Looking at the code, I’m not sure how the “Unit tests failed” message appears when fail_build is set to false. Here’s what the relevant portion of my Fastfile looks like:

    scan(
      scheme: ENV["SCAN_SCHEME"],
      workspace: ENV["SCAN_WORKSPACE"],
      configuration: 'Debug',
      output_types: '',
      device: ENV["DEVICE"] || 'iPhone 8 (12.0)',
      code_coverage: true,
      fail_build: false
    )

    trainer(
      output_directory: './fastlane/test_output',
      extension: '.junit',
      fail_build: false
    )

[jacobh0]

Having the same issue.. although Status 65 appears to be a fastlane bug with scan

[cnoon]

Same here @SlaunchaMan. I've dug through the trainer gem logic and I can't find where the fail_build option is actually used. It's certainly specified in options.rb, but it doesn't look like it's being pulled out. Design flaw?

Are you also using the fastlane-plugin-trainer like I am @SlaunchaMan? After more investigation, I think the issue is there. It's throwing a Unit tests failed error if any of the tests failed in trainer_action.rb. That's probably the core issue.

@KrauseFx does that line up? If yes, I'd be happy to put together a PR to fix the issue. I'd assume we'd just want to avoid throwing that error if fail_build is false.

[SlaunchaMan]

@cnoon Yes, I am using it with the Fastlane plugin. Fingers crossed on a fix! 😄

[cnoon]

Okay, looks like this issue was fixed in 8b8a755. What's super weird is that the fix is in master and what looks like would be 0.8.1, but that's definitely not in 0.8.1 when you look. Almost looks like the rubygem released as 0.8.1 is not correct.

[cnoon]

I bet if you pointed your Gemfile at the repo and not at the release it would work. I'm going to give that a go now. I've never released a gem before, but I wonder if it's possible to release the wrong commit. It doesn't appear that this project uses git tags for tracking.

[cnoon]

Okay, more updates. I was off on my versioning a bit mentioned above. It appears that the trainer gem has been released as 0.8.1, but the fastlane-plugin-trainer gem has not been released since April 2017 and is still at 0.4.0. That's why it does not contain the fixes. Essentially, we need a new tag cut of the fastlane-plugin-trainer gem and we'd be all good to go. @KrauseFx is that something you could help out with? Or someone else with access?

In the meantime @SlaunchaMan, you can work around the issue by updating your Pluginfile to this:

gem "fastlane-plugin-trainer", :git => "https://github.com/KrauseFx/trainer.git", :ref => "28ddc18b"

That will pick up the fail_build logic and stop failing if there are failed tests. I've verified that this does fix the issue locally. Good luck!

[KrauseFx]

Will push a new release, thanks

[KrauseFx]

New release is live, let me know if it works

[SlaunchaMan]

Will do!

[cnoon]

Awesome, thanks @KrauseFx! 🍻

[rist]

we still experience the same issue with fastlane-plugin-trainer 0.4.1 and Fastlane 2.108.0 - see also #13 (comment)

[silverhammermba]

Still seeing this issue as well. Looking in the plist file I see that xcode is outputting a FailureSummaries block for the entire test target, including the error message. Other test target summaries get output normally. Trainer should be able to generate an <error> block from this in the relevant test suite.

[1oo7]

Still seeing this issue. fastlane-plugin-trainer 0.4.1, fastlane 2.165.0