v0.9.6: documents, downloads, PDFs and the phone's own files

Four tools and a document writer, plus the storage half of the phone: until
now the picker's answer (a content:// URI) reached the engine as a path, and
it said "cannot read" -- the app had no way to hand a phone's file over.

- md_to_html (new, src/tools/web.c + cst_web_md_document()): markdown in, one
  self-contained HTML document out -- doctype, utf-8, viewport, escaped
  title, a small stylesheet; no script, no external request. It is written
  into the workspace and delivered as a file card (for_user, so it never
  enters a request), the way deliver_file does. The default name comes from
  the source file's stem or, for pasted markdown, from the title with what a
  path cannot carry turned into '-'. It has a preview, so an approval prompt
  says "render x -> y.html" before anything is written.
- download (src/tools/web.c): an http(s) URL into a file in the workspace,
  streamed with a 128 MB ceiling; a redirect chain, the same CA handling and
  the same refusals as web_fetch, but the bytes stay on disk (a half file is
  removed on any failure, including HTTP >= 400). The name comes from the URL
  when the caller names only a directory.
- read_pdf (src/tools/fs_tools.c): the text layer and page count of a PDF,
  which read_file refuses because the bytes are binary -- and a PDF is
  exactly what arrives from a phone. Same extractor an attachment uses, its
  own 4 MiB input ceiling, 256 KiB of text by default with a note when cut.
- session_search (new, src/tools/session_tools.c): every stored session --
  this one and all the ones before it -- searched for a substring, reporting
  the session's id, message count and title, then the matching lines with
  their number and role. Case-insensitive by default, capped at 20 matches.
- device_info (same file, perm_name "device"): version, system/arch/CPU
  count, workspace, sandbox, model and permissions, config/temp/program
  directories, pid -- so the model stops guessing where it runs.
- Tools 3 and 4 come with tests/test_tools_ext.c (also in both build files):
  the document is a whole page, a script tag and a javascript: link do not
  survive, a hostile title stays text and cannot write outside the workspace,
  the preview names the file, read_pdf/download refuse what they must (no
  network in a test), session_search finds a fixture session and says so when
  nothing matches, device_info names the workspace. README's tool table grew
  five rows.

- Android storage: the picker (SAF) answers with a content:// URI, and the
  engine opens files by path, so the app now copies what it points at into
  its own directory first -- app/android/src/xyz/whcself/cstagent/
  AttachFile.java, called through a new app::materializeAttachment()
  (ToolProvider.h: one API for every platform; a desktop strips file:// and
  refuses content:// with a reason). The copy needs no permission at all: the
  grant travels with the URI.
- "All files access" (MANAGE_EXTERNAL_STORAGE) is the other half: it is what
  lets the engine read a path in shared storage that a task names rather than
  picks. Android has no runtime dialog for it, so main.cpp opens the one
  Settings screen once (AttachFile.java's requestAllFilesAccess, with the
  all-apps screen as a fallback) and never insists: without it everything
  works, one picked file at a time. On the NX789J the screen came up, the
  app-op reads back "allow", and a relaunch no longer asks.
- Composer.qml hands a content:// URI through untouched (it is not a path,
  and decoding it would corrupt it); AgentCore::attachFile() turns whatever
  the picker gave into a path first.

Desktop: build clean, make check 36/36 (the new test included, also under
ASan), --uitest all green, --selftest and cst-app-selftest OK.
The APK builds and installs (86 MB, release-signed) with the new Java class
in its dex.

Version 0.9.6.

app/src/core/AgentCore.cpp

@@ -1034,11 +1034,19 @@ QString AgentCore::attachFile(const QString &path) {
     return QStringLiteral("no engine");
   if (path.isEmpty())
     return QStringLiteral("no file");
+  /* What a picker hands over is not always a path: on a phone it is a SAF
+   * content:// URI, and the app copies that into its own directory first (the
+   * engine opens files by path). A desktop passes the path through. */
+  QString err2;
+  const QString local = app::materializeAttachment(path, &err2);
+  if (local.isEmpty())
+    return err2.isEmpty() ? QStringLiteral("that file cannot be attached")
+                          : err2;
   char err[256] = "";
-  if (cst_agent_attach_file(m_agent, path.toUtf8().constData(), err,
+  if (cst_agent_attach_file(m_agent, local.toUtf8().constData(), err,
                             sizeof(err)) != 0)
     return utf8(err[0] ? err : "that file cannot be attached");
-  m_attachNames << QFileInfo(path).fileName();
+  m_attachNames << QFileInfo(local).fileName();
   emit pendingChanged();
   return QString();
 }

app/src/core/ToolProvider.h

@@ -78,6 +78,22 @@ QString holdForeground(const QString &text);
 /* Drop it again, at the end of the turn (a phone: the notification goes). */
 void releaseForeground();
 
+/* Turn what a file chooser handed over into a path the engine can open. A
+ * desktop picker gives a path and nothing happens; a phone's is a SAF
+ * content:// URI, which the app copies into its own directory first (the
+ * grant travels with the URI, so no permission is involved -- and without
+ * the copy the engine, which opens files by path, would fail with "cannot
+ * read"). Returns the path to read, or "" with *err filled. */
+QString materializeAttachment(const QString &pathOrUri, QString *err);
+/* Does this app have "all files access" (Android 11+)? A desktop always
+ * does. Without it the app still attaches whatever the picker handed it; what
+ * it cannot do is read shared storage by path. */
+bool hasAllFilesAccess();
+/* Ask for it, through the one screen Android offers (there is no runtime
+ * dialog for it). A desktop, a phone that already has it or one too old to
+ * know about it returns "" and nothing opens. */
+QString requestAllFilesAccess();
+
 }  // namespace app
 
 #endif /* CST_APP_TOOLPROVIDER_H */

app/src/main.cpp

@@ -257,6 +257,26 @@ int main(int argc, char **argv) {
     QtAndroidPrivate::requestPermission(
         QStringLiteral("android.permission.POST_NOTIFICATIONS"));
   }
+  /* "All files access" is the other half of storage: it is what lets the
+   * engine read a path in shared storage (a file the user names, not one the
+   * picker hands over -- that one is copied into the app either way, see
+   * AttachFile.java). Android has no runtime dialog for it, only the Settings
+   * screen, so it is opened once (a marker file remembers) and never
+   * insisted on: without it everything still works, one picked file at a
+   * time. The line below is what logcat shows when a device behaves oddly. */
+  const bool allFiles = app::hasAllFilesAccess();
+  qInfo("storage: all files access = %s", allFiles ? "yes" : "no");
+  if (!allFiles) {
+    const QString marker = androidRoot() + QStringLiteral("/.all-files-asked");
+    if (!QFileInfo::exists(marker)) {
+      const QString why = app::requestAllFilesAccess();
+      if (!why.isEmpty())
+        qWarning("all files access: %s", qUtf8Printable(why));
+      QFile asked(marker);
+      if (asked.open(QIODevice::WriteOnly))
+        asked.write("1\n");
+    }
+  }
 #endif
   if (parser.isSet(planOpt))
     core.setPlanMode(true);

app/src/platform/android/AndroidTools.cpp

@@ -27,6 +27,7 @@
 #include <QGuiApplication>
 #include <QJniObject>
 #include <QMetaObject>
+#include <QStandardPaths>
 #include <QString>
 #include <QStringList>
 #include <QThread>
@@ -662,6 +663,89 @@ void releaseForeground() {
   });
 }
 
+/* What the picker gave us -> a path the engine can open. A SAF URI is copied
+ * into the app's own cache (the URI's grant covers the read; no permission is
+ * involved), a plain path is left alone. */
+QString materializeAttachment(const QString &pathOrUri, QString *err) {
+  if (err)
+    err->clear();
+  if (pathOrUri.isEmpty())
+    return QString();
+  if (!pathOrUri.startsWith(QLatin1String("content://"))) {
+    QString p = pathOrUri;
+    if (p.startsWith(QLatin1String("file://")))
+      p = p.mid(7);
+    return p;
+  }
+  const QString dest = QStandardPaths::writableLocation(
+                           QStandardPaths::CacheLocation) +
+                       QStringLiteral("/picked");
+  QString result, failure;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid()) {
+      failure = QStringLiteral("no Android context to copy the file with");
+      return;
+    }
+    QJniObject out = QJniObject::callStaticObjectMethod(
+        "xyz/whcself/cstagent/AttachFile", "materialize",
+        "(Landroid/content/Context;Ljava/lang/String;Ljava/lang/String;)"
+        "Ljava/lang/String;",
+        ctx.object<jobject>(), QJniObject::fromString(pathOrUri).object<jstring>(),
+        QJniObject::fromString(dest).object<jstring>());
+    if (!out.isValid()) {
+      failure = QStringLiteral("the phone could not hand that file over");
+      return;
+    }
+    const QString s = out.toString();
+    if (s.startsWith(QLatin1String("error: ")))
+      failure = s.mid(7);
+    else if (!s.isEmpty())
+      result = s;
+    /* "" means it was a path all along; the caller keeps what it had */
+  });
+  if (!failure.isEmpty()) {
+    if (err)
+      *err = failure;
+    return QString();
+  }
+  return result.isEmpty() ? pathOrUri : result;
+}
+
+bool hasAllFilesAccess() {
+  bool granted = true;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid())
+      return;
+    granted = QJniObject::callStaticMethod<jboolean>(
+        "xyz/whcself/cstagent/AttachFile", "hasAllFilesAccess",
+        "(Landroid/content/Context;)Z", ctx.object<jobject>());
+  });
+  return granted;
+}
+
+QString requestAllFilesAccess() {
+  if (qEnvironmentVariableIsSet("CST_APP_NO_PERM"))
+    return QString(); /* a test run touches no settings */
+  QString err;
+  onGuiThread([&] {
+    QJniObject ctx = appContext();
+    if (!ctx.isValid())
+      return;
+    QJniObject out = QJniObject::callStaticObjectMethod(
+        "xyz/whcself/cstagent/AttachFile", "requestAllFilesAccess",
+        "(Landroid/content/Context;)Ljava/lang/String;",
+        ctx.object<jobject>());
+    if (out.isValid()) {
+      const QString s = out.toString();
+      if (!s.isEmpty())
+        err = s;
+    }
+  });
+  return err;
+}
+
 int platformRegisterAndroid(AgentCore *core) {
   if (!core)
     return 0;

app/src/platform/linux/NotifyTool.cpp

@@ -134,9 +134,29 @@ QString holdForeground(const QString &text) {
   (void)text;
   return QString();
 }
-
 void releaseForeground() {}
 
+/* A desktop picker hands over a path, so there is nothing to turn into one:
+ * a file:// URL loses its scheme and a content:// URI (a portal that never
+ * gave us a path) is refused with what the user should do instead. */
+QString materializeAttachment(const QString &pathOrUri, QString *err) {
+  QString s = pathOrUri;
+  if (s.startsWith(QLatin1String("file://")))
+    s = s.mid(7);
+  if (s.startsWith(QLatin1String("content://"))) {
+    if (err)
+      *err = QStringLiteral("that picker handed over a content:// URI this "
+                            "desktop cannot open; pick the file by path "
+                            "instead");
+    return QString();
+  }
+  return s;
+}
+
+/* Every desktop process reads the files its user can. */
+bool hasAllFilesAccess() { return true; }
+QString requestAllFilesAccess() { return QString(); }
+
 int platformRegisterLinux(AgentCore *core) {
   static const cst_tool_param_t params[CST_TOOL_MAX_PARAMS] = {
       {"message", "string", 1, "text of the notification"},