Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test note of iOS 17 with Xcode 15 #1752

Open
KazuCocoa opened this issue Jun 12, 2023 · 2 comments
Open

test note of iOS 17 with Xcode 15 #1752

KazuCocoa opened this issue Jun 12, 2023 · 2 comments
Labels
help wanted

Comments

@KazuCocoa
Copy link
Member

KazuCocoa commented Jun 12, 2023 •
edited
Loading

The list of tests we may need to check for ios 17 as this repository

npm run e2e-test:base

  • Need to check the faceID test app side
  4) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should accept matching fingerprint if faceID is enrolled:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:96:9)

  5) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should reject not matching fingerprint if faceID is enrolled:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:112:9)

  6) XCUITestDriver - faceID -
       faceID enrollment functional tests applied to FaceID sample app
         should enroll faceID and accept matching fingerprints then unenroll faceID and not be supported:
     Error: Element located by '-ios class chain:**/XCUIElementTypeStaticText[`label == "Face ID"`]' done not exist after 5000ms timeout
      at waitUntilExist (test/functional/basic/face-id-e2e-specs.js:64:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Context.<anonymous> (test/functional/basic/face-id-e2e-specs.js:137:9)

  8) XCUITestDriver - touchID -
       touchID enrollment functional tests applied to TouchId sample app
         "before each" hook for "should not support touchID if not enrolled":
     Error: Failed to create session.
An unknown server-side error occurred while processing the command. Original error: Could not create simulator with name 'appiumTest-C864A8C2-1155-41C1-83C4-8AAF6332483C-iPhone 8', device type id 'iPhone 8', with runtime ids 'com.apple.CoreSimulator.SimRuntime.iOS-17-0'
      at startWebDriverSession (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriver/build/utils.js:68:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Function.newSession (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriver/build/index.js:26:45)
      at async remote (file:///Users/kazu/GitHub/appium-xcuitest-driver/node_modules/webdriverio/build/index.js:48:22)
      at async initSession (test/functional/helpers/session.js:14:14)
      at async Context.<anonymous> (test/functional/basic/touch-id-e2e-specs.js:73:18)

npm run e2e-test:device

      1) should return recorded trace file on stop
2023-06-12T02:44:24.979Z INFO webdriver: COMMAND deleteSession()
2023-06-12T02:44:24.980Z INFO webdriver: [DELETE] http://127.0.0.1:4567/session/e1069e7e-9bbe-4091-b632-1f771f660ece
2023-06-12T02:44:25.003Z WARN webdriver: Request failed with status 404 due to A session is either terminated or not started
2023-06-12T02:44:25.003Z INFO webdriver: Retrying 1/1
2023-06-12T02:44:25.003Z INFO webdriver: [DELETE] http://127.0.0.1:4567/session/e1069e7e-9bbe-4091-b632-1f771f660ece
2023-06-12T02:44:25.007Z ERROR webdriver: Request failed with status 404 due to invalid session id: A session is either terminated or not started

e2e-test:driver


  1) XCUITestDriver
       should start and stop a session:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  2) XCUITestDriver
       reset
         with udid: uses sim and resets afterwards if resetOnSessionStartOnly is false:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  3) XCUITestDriver
       reset
         with udid booted: uses sim and leaves it afterwards:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

  4) XCUITestDriver
       reset
         with noReset set to true: leaves sim booted:
     Error: Timeout of 240000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/kazu/GitHub/appium-xcuitest-driver/test/functional/driver/driver-e2e-specs.js)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7)

e2e-test:long

ok

e2e-test:parallel

e2e-test:web

maybe need to update test code

  1) Safari - basics -
       init
         should start a session with custom init:

      AssertionError: expected 'Appium/welcome' to equal 'I am a page title'
      + expected - actual

      -Appium/welcome
      +I am a page title

      at Context.<anonymous> (test/functional/web/safari-basic-e2e-specs.js:45:26)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)

  2) iPhone Xs -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:134:21)

  3) iPhone Xs -
       should be able to tap on a button:
     AssertionError: expected '<html><head>\n  <title>I am a page ti…' to include 'Your comments: Hello'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/safari-nativewebtap-e2e-specs.js:145:43
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:143:21)

  4) iPad (10th generation) -
       should be able to bypass measuring the offset of banner:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:124:21)

  5) iPad (10th generation) -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:134:21)

  6) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:189:25)

  7) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling, when the url bar is present:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:67:19
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at wrapped (node_modules/asyncbox/lib/asyncbox.js:95:13)
      at retry (node_modules/asyncbox/lib/asyncbox.js:68:13)
      at retryInterval (node_modules/asyncbox/lib/asyncbox.js:105:10)
      at spinTitleEquals (test/functional/web/helpers.js:64:5)
      at Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:212:25)

e2e-test:native-web-tap

Test code needs to be updated

  1) iPhone Xs -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:145:11)

  2) iPhone Xs -
       should be able to tap on a button:
     AssertionError: expected '<html><head>\n  <title>I am a page ti…' to include 'Your comments: Hello'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/safari-nativewebtap-e2e-specs.js:162:31
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:160:11)

  3) iPad (10th generation) -
       should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:145:11)

  4) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:223:13)

  5) iPad (10th generation) -
       with tabs -
         should be able to tap on an element after scrolling, when the url bar is present:
     Error: Could not find expected title: 'Another Page: page 3'. Found: 'I am a page title'
      at /Users/kazu/GitHub/appium-xcuitest-driver/test/functional/web/helpers.js:53:13
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async wrapped (node_modules/asyncbox/build/lib/lib/asyncbox.js:95:13)
      at async retry (node_modules/asyncbox/build/lib/lib/asyncbox.js:68:13)
      at async retryInterval (node_modules/asyncbox/build/lib/lib/asyncbox.js:105:10)
      at async spinTitleEquals (test/functional/web/helpers.js:50:3)
      at async Context.<anonymous> (test/functional/web/safari-nativewebtap-e2e-specs.js:251:13)
@KazuCocoa KazuCocoa mentioned this issue Jun 12, 2023
Closed
@KazuCocoa
Copy link
Member Author

Real device set location should be via com.apple.instruments.server.services.LocationSimulation instead of com.apple.dt.simulatelocation over the remote server.

@mwakizaka
Copy link
Contributor

I'll try updating the tests if no problem

@mwakizaka mwakizaka mentioned this issue Jan 4, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@KazuCocoa @mwakizaka