Merge branch '8.4RC'

Owa compatibility for Friendica

See merge request hubzilla/core!2036

.gitlab-ci.yml

@@ -3,7 +3,7 @@
 # Use a prepared Hubzilla image to optimise pipeline duration
 # image: registry.gitlab.com/dawnbreak/hubzilla/core:php7.3
 
-image: php:8.0
+image: php:8.1
 
 stages:
   - test
@@ -96,8 +96,8 @@ before_script:
       - tests/results/
 
 
-# PHP8.0
-php8.0:
+# PHP8.1
+php8.1:
   <<: *job_definition_php
 
 # PHP8.0 with MySQL 5.7

CHANGELOG

@@ -1,3 +1,185 @@
+Hubzilla 8.4 (2023-05-17)
+	- Slightly rewrite Activity::store() to save a query
+	- Use act->objprop() in decode_note() to get activity values
+	- Make sure to re-discover a channel upon connecting if we have just an xchan but no hubloc
+	- Updated ES translations
+	- Updated RU translations
+	- Redesigned profile vcard to implement cover images
+	- Slightly improved discovery of AP quoted messages
+	- Updated bootstrap library
+	- Changed hashtag encoding to match mastodons expectations (issue #1750)
+	- Implement fedearted directory flags from trusted directory servers
+	- Use Activity::get_actor_hublocs() in Libzot::fetch_conversation() instead of custom queries
+	- Add the conversation endpoint
+	- Implement paginated fetch for zot requests in mod item
+	- Implement Zotconvo daemon to fetch conversations in the background
+	- Expose deleted channels to webfinger (otherwise we can not mark them deleted locally)
+	- Expire items in batches to prevent memory exhaustion
+	- Remove legacy zot compatibility tweaks
+	- Rewrite and simplify directory sync
+	- Improve potentially long running sql query finding the thread parent
+	- Implement option to override permissions for posts with mentions
+
+	Bugfixes
+	- Fix parent_mid and thr_parent not set correctly for response activities
+	- Fix query for postgres in attach_list_files()
+	- Fix item_url_replace() failing on arrays
+	- Fix OWA compatibility with friendica
+	- Fix regression in decoding the object type
+	- Fix xss vulnerability in justified gallery library
+	- Fix permissions not initialized after accepted follow request from AP
+	- Fix empty path passed to fopen()
+	- Fix % value in format_poll() not rounded
+	- Fix remove_all_xchan_resources() executed for local channels
+	- Fix likes not synced via libsync between clones
+
+	Addons
+	- Pubcrawl: fix xchan_url used as follow object instead of xchan_hash
+	- Channelreputation: moved to unmaintained repo due to duplication issues when storing data in pconfig
+	- Pubcrawl: add lemmy accept/reject follow quirks
+	- Diaspora: handle 4xx return codes in queue management
+	- Diaspora: fix fatal error in the case where parent_guid is set but empty
+	- Pubcrawl: if shared inbox delivery fails deliver to contacts instead of everybody we know from that site
+	- Articles: fix PHP error
+	- Diaspora: rewrite synced (from a clone) item author to primary
+	- Diaspora: improve reshare detection
+	- Diaspora: fix contact role not sets
+	- Pubcrawl: rewrite synced (from a clone) item author to primary
+	- Pubcrawl: if we receive a public message to the shared inbox deliver to following *and* directly addressed
+
+
+Hubzilla 8.2 (2023-03-19)
+	- Remove redundant untranslated htconfig templates
+	- Implement workaround for friendica image/attachment construct
+	- Implement notification exception for forum posts repeated by group actors
+	- Updated es translations
+	- Rewrite totp multifactor auth and move it from addon to core (ported from streams)
+	- Implement page load progress indicator for pwa standalone mode
+	- Improved mod HQ logic to save some DB lookups
+	- Default owner_xchan to sender in Activity::store() except for fetched items
+	- Improved contact search
+	- Do not force new thread if announce comes from group actor
+	- Disable oembed in notes app
+	- Improved queue handling
+	- Slightly restructure conv item head
+	- Implemet author images in HQ widget
+	- Implement author filter in HQ widget
+	- Add config option db_skip_locked_supported which should be set if the DB backend supports skip locked
+	- Updated simple pie library
+	- Updated smarty library
+	- Updated forkawesome library
+	- Updated fullcalendar library
+	- Implement native dark/light mode in favour of dark/light schema
+	- Updated bootstrap library
+
+	Bugfixes
+	- Fix channel calendar location html not interpreted - issue #1728
+	- Fix max oembed size check if content length header is an array
+	- Fix group actor wall posts turned into direct messages when posting article or card
+	- Fix non zot profile mentions using zrl tag
+	- Fix message filter > and < not decoded
+	- Fix deleted hublocs included in author lookup
+	- Fix some more PHP warnings
+	- Fix syntax for get_compund_property()
+	- Fix xchan_pubforum not set on actor updates
+	- Fix duplicate directory entries for xchans with bogus second primary entry
+	- Fix missing created timestamp in like activity - issue #1729
+	- Fix messages not signed when posting from non primary locations
+	- Fix pinned post regression
+	- Fix autocomplete items not selectable on mobile
+	- Fix hubloc confusion in Libzot::fetch_conversation()
+	- Fix valid duplicate notification entries deduplicated
+	- Fix ics file import
+	- Fix recipients deduplication in some places
+	- Fix hashtag encoded with href instead of id
+
+	Addons
+	- Pubcrawl: re-implement encrypted content which got lost in transition
+	- Pubcrawl: flag fetched items
+	- Totp: removed from addons in favor of implementation in core
+	- Pubcrawl: fix delivery issues for relayed activities
+	- Cart: fix missing bootstrap namespace
+	- Gallery: do not inject gallery code in webpages
+	- Pubcrawl: fix sql error if mentions string is empty
+	- Cart: fix paypal order hash size
+
+
+Hubzilla 8.0 (2023-01-13)
+	- Updated ru strings
+	- Implement ability to edit addon PDL with pdledit_gui and pdledit
+	- Implement queueworker auto sleep time
+	- Always add sys channel to receivers for public items
+	- Improved server sent events loop
+	- Exclude not tagable xchan networks handle_tag()
+	- Add mark html tag to the html2bbcode parser
+	- Embed or attach uploaded files depending on file type
+	- Changed logic to enable site only public stream - please revisit admin setting
+	- Updated nb-no strings
+	- Moved queueworker to core
+	- Provide a PDL file for mod invite and set the profile
+	- Disabled outdated context help
+	- Mark unseen items seen after a certain amount of time (default 90 days)
+	- Slightly restructure lib webfinger
+	- Improved redbasic dark schema
+	- Implement unseen thread and unseen items count in HQ widget
+	- In unseen items notifications just count the first 99 and display 99+ if there are more
+
+	Bugfixes
+	- Fix race conditions when processing multiple choice polls
+	- Fix affinity slider updates - issue #1714
+	- Fix issue where accounts created with a did2 could change their password
+	- Fix no message displayed if file not found in mod cloud
+	- Fix pubstream tagcloud unthreaded view
+	- Fix french invite template
+	- Fix comment permission issue in site only public stream
+	- Fix privacy tag not referring to latest hubloc entry
+	- Fix selected item not highlighted
+	- Fix poll issue when special chars and spaces are involved
+	- Fix prune_hub_reinstalls() to only care about zot6 hublocs
+	- Fix redis session PHP warnings
+	- Fix remote visitor photo uploads
+	- Fix mod shredwithme not showing any items
+	- Fix api_auth not referring to latest hubloc entry
+	- Fix category widget URLs
+	- Fix poller always polling broken feeds
+	- Fix sse event always triggered if public stream notification are off or the app is not installed
+	- Fix more PHP8 warnings
+	- Fix wrong variable in tag_deliver()
+	- Fix mod follow use trim() after punify() - issue #1698
+
+	Addons
+	- Cards: update widget info
+	- Articles: update widget info
+	- Wiki: update widget info
+	- Sse: filter apporvals
+	- Cards: use html_entity_decode before purify and html2plain for summary
+	- Articles: use html_entity_decode before purify and html2plain for summary
+	- Wiki: use html_entity_decode before purify and html2plain for summary
+	- Cards: fix comments not displayed if filter by category
+	- Articles: fix comments not displayed if filter by category
+	- Sse: respect the site public stream setting
+	- Queueworker: moved to core
+	- Pubcrawl: move db query out of foreach loop
+	- Pubcrawl: add mentions and thread owner to recipients for public items
+
+
+Hubzilla 7.8.7 (2022-12-03)
+	- Fix regression when adding feed contacts
+	- Fix regression new channel calendar event not created
+
+
+Hubzilla 7.8.6 (2022-11-14)
+	- Fix typo in boot.php
+
+
+Hubzilla 7.8.5 (2022-11-13)
+	- Fix outbound edit activity not of type update
+	- Fix mod display not falling through to fetch public item
+	- Fix more PHP warnings
+	- Fix regression in items_fetch() which resulted in empty atom feed
+	- Pubcrawl: cleanup and slightly restructre mod inbox
+
+
 Hubzilla 7.8.4 (2022-11-09)
 	- Fix new uuid created when editing a post
 

Zotlabs/Daemon/Addon.php

@@ -7,6 +7,7 @@ class Addon {
 	static public function run($argc, $argv) {
 
 		call_hooks('daemon_addon', $argv);
+		return;
 
 	}
 

Zotlabs/Daemon/Cache_embeds.php

@@ -19,8 +19,10 @@ class Cache_embeds {
 		$item = $c[0];
 
 		// bbcode conversion by default processes embeds that aren't already cached.
-		// Ignore the returned html output. 
+		// Ignore the returned html output.
 
 		bbcode($item['body']);
+
+		return;
 	}
 }

Zotlabs/Daemon/Cache_query.php

@@ -24,7 +24,7 @@ class Cache_query {
 
 		array_shift($argv);
 		array_shift($argv);
-		
+
 		$arr = json_decode(base64_decode($argv[0]), true);
 
 		$r = call_user_func_array('q', $arr);
@@ -32,5 +32,7 @@ class Cache_query {
 			Cache::set($key, serialize($r));
 
 		del_config('procid', $key);
+
+		return;
 	}
 }

Zotlabs/Daemon/Cli_suggest.php

@@ -9,6 +9,7 @@ class Cli_suggest {
 	static public function run($argc,$argv) {
 
 		update_suggestions();
+		return;
 
 	}
 }

Zotlabs/Daemon/Convo.php

@@ -30,6 +30,7 @@ class Convo {
 			intval($channel_id),
 			dbesc($contact_hash)
 		);
+
 		if (!$r) {
 			return;
 		}
@@ -40,19 +41,26 @@ class Convo {
 
 		$messages = $obj->get();
 
-		if ($messages) {
-			foreach ($messages as $message) {
-				if (is_string($message)) {
-					$message = Activity::fetch($message, $channel);
-				}
-				// set client flag because comments will probably just be objects and not full blown activities
-				// and that lets us use implied_create
-				$AS = new ActivityStreams($message);
-				if ($AS->is_valid() && is_array($AS->obj)) {
-					$item = Activity::decode_note($AS);
-					Activity::store($channel, $contact['abook_xchan'], $AS, $item);
-				}
+		if (!$messages) {
+			return;
+		}
+
+		foreach ($messages as $message) {
+			if (is_string($message)) {
+				$message = Activity::fetch($message, $channel);
+			}
+
+			// set client flag because comments will probably just be objects and not full blown activities
+			// and that lets us use implied_create
+			$AS = new ActivityStreams($message);
+			if ($AS->is_valid() && is_array($AS->obj)) {
+				$item = Activity::decode_note($AS);
+				$item['item_fetched'] = true;
+				Activity::store($channel, $contact['abook_xchan'], $AS, $item);
 			}
 		}
+
+		return;
+
 	}
 }

Zotlabs/Daemon/Cron.php

@@ -3,6 +3,7 @@
 namespace Zotlabs\Daemon;
 
 use Zotlabs\Lib\Libsync;
+use Zotlabs\Lib\Libzotdir;
 
 class Cron {
 
@@ -19,6 +20,7 @@ class Cron {
 			}
 		}
 
+/*
 		// Check for a lockfile.  If it exists, but is over an hour old, it's stale.  Ignore it.
 		$lockfile = 'store/[data]/cron';
 		if ((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600))
@@ -30,9 +32,21 @@ class Cron {
 		// Create a lockfile.  Needs two vars, but $x doesn't need to contain anything.
 		$x = '';
 		file_put_contents($lockfile, $x);
+*/
 
 		logger('cron: start');
 
+		// If this is a directory server, request a sync with an upstream
+		// directory at least once a day, up to once every poll interval.
+		// Pull remote changes and push local changes.
+		// potential issue: how do we keep from creating an endless update loop?
+
+		$dirmode = get_config('system', 'directory_mode');
+
+		if ($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
+			Libzotdir::sync_directories($dirmode);
+		}
+
 		// run queue delivery process in the background
 
 		Master::Summon(array('Queue'));
@@ -50,7 +64,7 @@ class Cron {
 		require_once('include/account.php');
 		remove_expired_registrations();
 
-		$interval = get_config('system', 'delivery_interval', 3);
+		$interval = get_config('queueworker', 'queue_interval', 500000);
 
 		// expire any expired items
 
@@ -65,8 +79,10 @@ class Cron {
 				if ($rr['item_wall']) {
 					// The notifier isn't normally invoked unless item_drop is interactive.
 					Master::Summon(['Notifier', 'drop', $rr['id']]);
-					if ($interval)
-						@time_sleep_until(microtime(true) + (float)$interval);
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
 		}
@@ -96,8 +112,10 @@ class Cron {
 		if ($r) {
 			foreach ($r as $rr) {
 				Master::Summon(array('Directory', $rr['channel_id'], 'force'));
-				if ($interval)
-					@time_sleep_until(microtime(true) + (float)$interval);
+
+				if ($interval) {
+					usleep($interval);
+				}
 			}
 		}
 
@@ -151,8 +169,10 @@ class Cron {
 						);
 					}
 					Master::Summon(array('Notifier', 'wall-new', $rr['id']));
-					if ($interval)
-						@time_sleep_until(microtime(true) + (float)$interval);
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
 		}
@@ -203,10 +223,10 @@ class Cron {
 		}
 
 
-		// pull in some public posts
+		// pull in some public posts if allowed
 
-		$disable_discover_tab = get_config('system', 'disable_discover_tab') || get_config('system', 'disable_discover_tab') === false;
-		if (!$disable_discover_tab)
+		$disable_externals = get_config('system', 'disable_discover_tab') || get_config('system', 'disable_discover_tab') === false || get_config('system', 'site_firehose');
+		if (!$disable_externals)
 			Master::Summon(['Externals']);
 
 		$restart = false;
@@ -228,7 +248,7 @@ class Cron {
 		set_config('system', 'lastcron', datetime_convert());
 
 		//All done - clear the lockfile
-		@unlink($lockfile);
+		//@unlink($lockfile);
 
 		return;
 	}

Zotlabs/Daemon/Cron_daily.php

@@ -49,6 +49,22 @@ class Cron_daily {
 			dbesc('sse_id.%')
 		);
 
+		// Mark items seen after X days (default 90)
+
+		$r = dbq("select channel_id from channel where channel_removed = 0");
+		if ($r) {
+			foreach ($r as $rr) {
+				$mark_seen_days = get_pconfig($rr['channel_id'], 'system', 'mark_seen_days', 90);
+				q("UPDATE item SET item_unseen = 0 WHERE
+					uid = %d AND item_unseen = 1
+					AND created < %s - INTERVAL %s",
+					intval($rr['channel_id']),
+					db_utcnow(),
+					db_quoteinterval($mark_seen_days . ' DAY')
+				);
+			}
+		}
+
 		// Clean up emdedded content cache
 		q("DELETE FROM cache WHERE updated < %s - INTERVAL %s",
 			db_utcnow(),
@@ -78,17 +94,6 @@ class Cron_daily {
 		// expire any expired accounts
 		downgrade_accounts();
 
-		// If this is a directory server, request a sync with an upstream
-		// directory at least once a day, up to once every poll interval.
-		// Pull remote changes and push local changes.
-		// potential issue: how do we keep from creating an endless update loop?
-
-		$dirmode = get_config('system', 'directory_mode');
-
-		if ($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
-			Libzotdir::sync_directories($dirmode);
-		}
-
 		Master::Summon(array('Expire'));
 		Master::Summon(array('Cli_suggest'));
 
@@ -103,5 +108,7 @@ class Cron_daily {
 		/**
 		 * End Cron Daily
 		 */
+
+		 return;
 	}
 }

Zotlabs/Daemon/Cron_weekly.php

@@ -22,12 +22,12 @@ class Cron_weekly {
 
 		mark_orphan_hubsxchans();
 
-		// Find channels that were removed in the last three weeks, but 
+		// Find channels that were removed in the last three weeks, but
 		// haven't been finally cleaned up. These should be older than 10
-		// days to ensure that "purgeall" messages have gone out or bounced 
-		// or timed out. 
+		// days to ensure that "purgeall" messages have gone out or bounced
+		// or timed out.
 
-		$r = q("select channel_id from channel where channel_removed = 1 and 
+		$r = q("select channel_id from channel where channel_removed = 1 and
 			channel_deleted >  %s - INTERVAL %s and channel_deleted < %s - INTERVAL %s",
 			db_utcnow(), db_quoteinterval('21 DAY'),
 			db_utcnow(), db_quoteinterval('10 DAY')
@@ -59,5 +59,6 @@ class Cron_weekly {
 		 * End Cron Weekly
 		 */
 
+		return;
 	}
-}
+}

Zotlabs/Daemon/Deliver.php

@@ -28,6 +28,8 @@ class Deliver {
 
 		}
 
+		return;
+
 	}
 
 }

Zotlabs/Daemon/Deliver_hooks.php

@@ -12,8 +12,12 @@ class Deliver_hooks {
 		$r = q("select * from item where id = '%d'",
 			intval($argv[1])
 		);
-		if ($r)
+
+		if ($r) {
 			call_hooks('notifier_normal', $r[0]);
+		}
+
+		return;
 
 	}
 }

Zotlabs/Daemon/Directory.php

@@ -97,5 +97,7 @@ class Directory {
 		if ($pushall) {
 			Master::Summon(array('Notifier', 'refresh_all', $channel['channel_id']));
 		}
+
+		return;
 	}
 }

Zotlabs/Daemon/Expire.php

@@ -32,7 +32,6 @@ class Expire {
 		}
 
 		// physically remove anything that has been deleted for more than two months
-		/** @FIXME - this is a wretchedly inefficient query */
 
 		q("delete from item where item_pending_remove = 1 and changed < %s - INTERVAL %s",
 			db_utcnow(),
@@ -59,8 +58,8 @@ class Expire {
 					continue;
 
 				// service class default (if non-zero) over-rides the site default
-
 				$service_class_expire = service_class_fetch($rr['channel_id'], 'expire_days');
+
 				if (intval($service_class_expire))
 					$channel_expire = $service_class_expire;
 				else
@@ -85,7 +84,6 @@ class Expire {
 
 			// this should probably just fetch the channel_expire_days from the sys channel,
 			// but there's no convenient way to set it.
-
 			$expire_days = get_config('system', 'sys_expire_days');
 			if ($expire_days === false)
 				$expire_days = 30;
@@ -96,12 +94,15 @@ class Expire {
 
 			logger('Expire: sys interval: ' . $expire_days, LOGGER_DEBUG);
 
-			if ($expire_days)
+			if ($expire_days) {
 				item_expire($x['channel_id'], $expire_days, $commented_days);
+			}
 
 			logger('Expire: sys: done', LOGGER_DEBUG);
 		}
 
 		del_config('procid', 'expire');
+
+		return;
 	}
 }

Zotlabs/Daemon/Externals.php

@@ -19,6 +19,7 @@ class Externals {
 		$importer = get_sys_channel();
 		$total    = 0;
 		$attempts = 0;
+		$url = '';
 
 		logger('externals: startup', LOGGER_DEBUG);
 
@@ -67,9 +68,8 @@ class Externals {
 					datetime_convert('UTC', 'UTC', 'now - 30 days')
 				);
 
-				$contact = $r[0];
-
-				if ($contact) {
+				if ($r) {
+					$contact = $r[0];
 					$url = $contact['hubloc_id_url'];
 				}
 			}
@@ -143,7 +143,8 @@ class Externals {
 						$AS = new ActivityStreams($message);
 						if ($AS->is_valid() && is_array($AS->obj)) {
 							$item = Activity::decode_note($AS);
-							Activity::store($importer, $contact['abook_xchan'], $AS, $item);
+							$item['item_fetched'] = true;
+							Activity::store($importer, $contact['hubloc_hash'], $AS, $item);
 							$total++;
 						}
 					}

Zotlabs/Daemon/Importdoc.php

@@ -11,6 +11,8 @@ class Importdoc {
 
 		self::update_docs_dir('doc/*');
 
+		return;
+
 	}
 
 	static public function update_docs_dir($s) {

Zotlabs/Daemon/Master.php

@@ -2,6 +2,8 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\QueueWorker;
+
 if (array_search(__file__, get_included_files()) === 0) {
 	require_once('include/cli_startup.php');
 	array_shift($argv);
@@ -9,6 +11,7 @@ if (array_search(__file__, get_included_files()) === 0) {
 
 	if ($argc)
 		Master::Release($argc, $argv);
+
 	return;
 }
 
@@ -16,6 +19,10 @@ if (array_search(__file__, get_included_files()) === 0) {
 class Master {
 
 	static public function Summon($arr) {
+
+		QueueWorker::Summon($arr);
+		return;
+/*
 		$hookinfo = [
 			'argv' => $arr
 		];
@@ -32,11 +39,15 @@ class Master {
 
 		$phpbin = get_config('system', 'phpbin', 'php');
 		proc_run($phpbin, 'Zotlabs/Daemon/Master.php', $arr);
+*/
 	}
 
 	static public function Release($argc, $argv) {
 		cli_startup();
 
+		QueueWorker::Release($argv);
+		return;
+/*
 		$hookinfo = [
 			'argv' => $argv
 		];
@@ -54,5 +65,6 @@ class Master {
 		logger('Master: release: ' . json_encode($argv), LOGGER_ALL, LOG_DEBUG);
 		$cls = '\\Zotlabs\\Daemon\\' . $argv[0];
 		$cls::run($argc, $argv);
+*/
 	}
 }

Zotlabs/Daemon/Notifier.php

@@ -79,10 +79,7 @@ class Notifier {
 	static public $encoded_item = null;
 	static public $channel = null;
 	static public $private = false;
-	// $fragment can contain additional info to omit de-duplication in the queueworker.
-	// E.g. if an item is updated many times in a row from different sources (multiple vote updates) the
-	// update source mid or a timestamp or random string can be added.
-	static public $fragment = null;
+
 
 	static public function run($argc, $argv) {
 
@@ -107,7 +104,6 @@ class Notifier {
 		self::$encoded_item = null;
 		self::$channel      = null;
 		self::$private      = false;
-		self::$fragment     = null;
 
 		$sys         = get_sys_channel();
 		$normal_mode = true;
@@ -233,8 +229,6 @@ class Notifier {
 
 			// Fetch the target item
 
-			self::$fragment = $argv[3] ?? '';
-
 			$r = q("SELECT * FROM item WHERE id = %d AND parent != 0",
 				intval($item_id)
 			);
@@ -415,8 +409,6 @@ class Notifier {
 				self::$private    = false;
 				self::$recipients = collect_recipients($parent_item, self::$private);
 
-				// FIXME add any additional recipients such as mentions, etc.
-
 				if ($top_level_post) {
 					// remove clones who will receive the post via sync
 					self::$recipients = array_values(array_diff(self::$recipients, [$target_item['owner_xchan']]));
@@ -509,6 +501,7 @@ class Notifier {
 		// public posts won't make it to the local public stream unless there's a recipient on this site.
 		// This code block sees if it's a public post and localhost is missing, and if so adds an entry for the local sys channel to the $hubs list
 
+		/* sys channel is now added in collect recipients
 		if (!self::$private) {
 			$found_localhost = false;
 			if ($hubs) {
@@ -529,6 +522,7 @@ class Notifier {
 				}
 			}
 		}
+		*/
 
 		if (!$hubs) {
 			logger('notifier: no hubs', LOGGER_NORMAL, LOG_NOTICE);
@@ -699,7 +693,7 @@ class Notifier {
 			// This wastes a process if there are no delivery hooks configured, so check this before launching the new process
 			$x = q("select * from hook where hook = 'notifier_normal'");
 			if ($x) {
-				Master::Summon(['Deliver_hooks', $target_item['id'], self::$fragment]);
+				Master::Summon(['Deliver_hooks', $target_item['id']]);
 			}
 		}
 

Zotlabs/Daemon/Onedirsync.php

@@ -15,61 +15,45 @@ class Onedirsync {
 			$update_id = intval($argv[1]);
 
 		if (!$update_id) {
-			logger('onedirsync: no update');
+			logger('onedirsync: no update id');
 			return;
 		}
 
-		$r = q("select * from updates where ud_id = %d limit 1",
+		$r = q("select * from updates where ud_id = %d",
 			intval($update_id)
 		);
 
-		if (!$r)
-			return;
-
-		if (($r[0]['ud_flags'] & UPDATE_FLAGS_UPDATED) || (!$r[0]['ud_addr']))
-			return;
-
-		// Have we probed this channel more recently than the other directory server
-		// (where we received this update from) ?
-		// If we have, we don't need to do anything except mark any older entries updated
-
-		$x = q("select * from updates where ud_addr = '%s' and ud_date > '%s' and ( ud_flags & %d )>0 order by ud_date desc limit 1",
-			dbesc($r[0]['ud_addr']),
-			dbesc($r[0]['ud_date']),
-			intval(UPDATE_FLAGS_UPDATED)
-		);
-		if ($x) {
-			q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and ( ud_flags & %d ) = 0 and ud_date != '%s'",
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($r[0]['ud_addr']),
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($x[0]['ud_date'])
-			);
+		if (!$r) {
+			logger('onedirsync: update id not found');
 			return;
 		}
 
 		// ignore doing an update if this ud_addr refers to a known dead hubloc
 
-		$h = q("select * from hubloc where hubloc_addr = '%s'",
+		$h = q("select * from hubloc where hubloc_id_url = '%s' order by hubloc_id desc",
 			dbesc($r[0]['ud_addr']),
 		);
 
 		$h = Libzot::zot_record_preferred($h);
 
 		if (($h) && (($h['hubloc_status'] & HUBLOC_OFFLINE) || $h['hubloc_deleted'] || $h['hubloc_error'])) {
-			q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and ( ud_flags & %d ) = 0 ",
-				intval(UPDATE_FLAGS_DELETED),
-				dbesc($r[0]['ud_addr']),
-				intval(UPDATE_FLAGS_UPDATED)
-			);
-			return;
+
+			// 2023-04-12: Try to update anyway since the info is not always correct
+			// This might change after all directory servers run the new code.
+
+			// q("update updates set ud_flags = 9 where ud_hash = '%s' and ud_flags != 9",
+			//	dbesc($r[0]['ud_hash'])
+			//);
+
+			// return;
 		}
 
 		// we might have to pull this out some day, but for now update_directory_entry()
 		// runs zot_finger() and is kind of zot specific
 
-		if ($h && $h['hubloc_network'] !== 'zot6')
+		if ($h && $h['hubloc_network'] !== 'zot6') {
 			return;
+		}
 
 		Libzotdir::update_directory_entry($r[0]);
 

Zotlabs/Daemon/Onepoll.php

@@ -30,14 +30,10 @@ class Onepoll {
 			$sql_extra = ' and abook_feed = 0 ';
 		}
 
-		$contacts = q("SELECT abook.*, xchan.*, account.*
-			FROM abook LEFT JOIN account on abook_account = account_id left join xchan on xchan_hash = abook_xchan
-			where abook_id = %d $sql_extra
-			and abook_pending = 0 and abook_archived = 0 and abook_blocked = 0 and abook_ignored = 0
-			AND (( account_flags = %d ) OR ( account_flags = %d )) limit 1",
-			intval($contact_id),
-			intval(ACCOUNT_OK),
-			intval(ACCOUNT_UNVERIFIED)
+		$contacts = q("SELECT abook.*, xchan.* FROM abook
+			LEFT JOIN xchan ON xchan_hash = abook_xchan
+			WHERE abook_id = %d",
+			intval($contact_id)
 		);
 
 		if (!$contacts) {
@@ -64,12 +60,21 @@ class Onepoll {
 		if ($contact['xchan_network'] === 'rss') {
 			logger('onepoll: processing feed ' . $contact['xchan_name'], LOGGER_DEBUG);
 			$alive = handle_feed($importer['channel_id'], $contact_id, $contact['xchan_hash']);
-			if ($alive) {
-				q("update abook set abook_connected = '%s' where abook_id = %d",
+
+			if (!$alive) {
+				q("update abook set abook_updated = '%s' where abook_id = %d",
 					dbesc(datetime_convert()),
 					intval($contact['abook_id'])
 				);
+				return;
 			}
+
+			q("update abook set abook_updated = '%s', abook_connected = '%s' where abook_id = %d",
+				dbesc(datetime_convert()),
+				dbesc(datetime_convert()),
+				intval($contact['abook_id'])
+			);
+
 			return;
 		}
 
@@ -163,6 +168,7 @@ class Onepoll {
 							$AS = new ActivityStreams($message);
 							if ($AS->is_valid() && is_array($AS->obj)) {
 								$item = Activity::decode_note($AS);
+								$item['item_fetched'] = true;
 								Activity::store($importer, $contact['abook_xchan'], $AS, $item);
 							}
 						}

Zotlabs/Daemon/Poller.php

@@ -17,21 +17,7 @@ class Poller {
 			}
 		}
 
-		$interval = intval(get_config('system', 'poll_interval'));
-		if (!$interval)
-			$interval = ((get_config('system', 'delivery_interval') === false) ? 3 : intval(get_config('system', 'delivery_interval')));
-
-		// Check for a lockfile.  If it exists, but is over an hour old, it's stale.  Ignore it.
-		$lockfile = 'store/[data]/poller';
-		if ((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600))
-			&& (!get_config('system', 'override_poll_lockfile'))) {
-			logger("poller: Already running");
-			return;
-		}
-
-		// Create a lockfile.  Needs two vars, but $x doesn't need to contain anything.
-		$x = '';
-		file_put_contents($lockfile, $x);
+		$interval = get_config('queueworker', 'queue_interval', 500000);
 
 		logger('poller: start');
 
@@ -63,6 +49,11 @@ class Poller {
 			: ''
 		);
 
+		$allow_feeds = get_config('system', 'feed_contacts');
+		if(!$allow_feeds) {
+			$sql_extra .= ' and abook_feed = 0 ';
+		}
+
 		$randfunc = db_getfunc('RAND');
 
 		$contacts = q("SELECT abook.abook_updated, abook.abook_connected, abook.abook_feed,
@@ -72,7 +63,7 @@ class Poller {
 			account.account_lastlog, account.account_flags
 			FROM abook LEFT JOIN xchan on abook_xchan = xchan_hash
 			LEFT JOIN account on abook_account = account_id
-			where abook_self = 0
+			where abook_self = 0 and abook_pending = 0 and abook_archived = 0 and abook_blocked = 0 and abook_ignored = 0
 			$sql_extra
 			AND (( account_flags = %d ) OR ( account_flags = %d )) $abandon_sql ORDER BY $randfunc",
 			intval(ACCOUNT_OK),
@@ -93,13 +84,24 @@ class Poller {
 						$min = intval(get_config('system', 'minimum_feedcheck_minutes'));
 					if (!$min)
 						$min = 60;
-					$x = datetime_convert('UTC', 'UTC', "now - $min minutes");
-					if ($c < $x) {
-						Master::Summon(['Onepoll', $contact['abook_id']]);
-						if ($interval)
-							@time_sleep_until(microtime(true) + (float)$interval);
+
+					if ($t !== $c) {
+						// if the last fetch failed only attempt fetch once a day
+						$min = 60 * 24;
 					}
+
+					$x = datetime_convert('UTC', 'UTC', "now - $min minutes");
+
+					if ($t < $x) {
+						Master::Summon(['Onepoll', $contact['abook_id']]);
+
+						if ($interval) {
+							usleep($interval);
+						}
+					}
+
 					continue;
+
 				}
 
 				if ($contact['xchan_network'] !== 'zot6')
@@ -158,20 +160,22 @@ class Poller {
 					continue;
 
 				Master::Summon(['Onepoll', $contact['abook_id']]);
-				if ($interval)
-					@time_sleep_until(microtime(true) + (float)$interval);
 
+				if ($interval) {
+					usleep($interval);
+				}
 			}
 		}
 
 		$dirmode = intval(get_config('system', 'directory_mode'));
 
 		if ($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
-			$r = q("SELECT u.ud_addr, u.ud_id, u.ud_last FROM updates AS u INNER JOIN (SELECT ud_addr, max(ud_id) AS ud_id FROM updates WHERE ( ud_flags & %d ) = 0 AND ud_addr != '' AND ( ud_last <= '%s' OR ud_last > %s - INTERVAL %s ) GROUP BY ud_addr) AS s ON s.ud_id = u.ud_id ",
-				intval(UPDATE_FLAGS_UPDATED),
+			$r = q("SELECT * FROM updates WHERE ud_update = 1 AND (ud_last = '%s' OR ud_last > %s - INTERVAL %s)",
 				dbesc(NULL_DATE),
-				db_utcnow(), db_quoteinterval('7 DAY')
+				db_utcnow(),
+				db_quoteinterval('7 DAY')
 			);
+
 			if ($r) {
 				foreach ($r as $rr) {
 
@@ -181,19 +185,18 @@ class Poller {
 					if ($rr['ud_last'] > NULL_DATE)
 						if ($rr['ud_last'] > datetime_convert('UTC', 'UTC', 'now - 1 day'))
 							continue;
+
 					Master::Summon(['Onedirsync', $rr['ud_id']]);
-					if ($interval)
-						@time_sleep_until(microtime(true) + (float)$interval);
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
 		}
 
 		set_config('system', 'lastpoll', datetime_convert());
 
-		//All done - clear the lockfile
-
-		@unlink($lockfile);
-
 		return;
 	}
 }

Zotlabs/Daemon/Queue.php

@@ -7,77 +7,57 @@ use Zotlabs\Lib\Queue as LibQueue;
 class Queue {
 
 	static public function run($argc, $argv) {
-
-		require_once('include/items.php');
-		require_once('include/bbcode.php');
-
-		if ($argc > 1)
-			$queue_id = $argv[1];
-		else
-			$queue_id = EMPTY_STR;
+		$queue_id = ($argc > 1) ? $argv[1] : '';
 
 		logger('queue: start');
 
 		// delete all queue items more than 3 days old
 		// but first mark these sites dead if we haven't heard from them in a month
 
-		$r = q("select outq_posturl from outq where outq_created < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('3 DAY')
+		$oldqItems = q("select outq_posturl, outq_hash from outq where outq_created < %s - INTERVAL %s",
+			db_utcnow(),
+			db_quoteinterval('3 DAY')
 		);
-		if ($r) {
-			foreach ($r as $rr) {
-				$h       = parse_url($rr['outq_posturl']);
-				$desturl = $h['scheme'] . '://' . $h['host'] . (isset($h['port']) ? ':' . $h['port'] : '');
+
+		if ($oldqItems) {
+			foreach ($oldqItems as $qItem) {
+				$h = parse_url($qItem['outq_posturl']);
+				$site_url = $h['scheme'] . '://' . $h['host'] . ((!empty($h['port'])) ? ':' . $h['port'] : '');
 				q("update site set site_dead = 1 where site_dead = 0 and site_url = '%s' and site_update < %s - INTERVAL %s",
-					dbesc($desturl),
-					db_utcnow(), db_quoteinterval('1 MONTH')
+					dbesc($site_url),
+					db_utcnow(),
+					db_quoteinterval('1 MONTH')
 				);
 			}
+
+			$old_hashes = ids_to_querystr($oldqItems, 'outq_hash', true);
+
+			logger('Removing ' . count($oldqItems) . ' old queue entries');
+			dbq("DELETE FROM outq WHERE outq_hash IN ($old_hashes)");
+
 		}
 
-		q("DELETE FROM outq WHERE outq_created < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('3 DAY')
-		);
+		$deliveries = [];
 
 		if ($queue_id) {
-			$r = q("SELECT * FROM outq WHERE outq_hash = '%s' LIMIT 1",
+			$qItems = q("SELECT * FROM outq WHERE outq_hash = '%s' LIMIT 1",
 				dbesc($queue_id)
 			);
+			logger('queue deliver: ' . $qItems[0]['outq_hash'] . ' to ' . $qItems[0]['outq_posturl'], LOGGER_DEBUG);
+			LibQueue::deliver($qItems[0]);
 		}
 		else {
-
-			// For the first 12 hours we'll try to deliver every 15 minutes
-			// After that, we'll only attempt delivery once per hour.
-			// This currently only handles the default queue drivers ('zot' or '') which we will group by posturl
-			// so that we don't start off a thousand deliveries for a couple of dead hubs.
-			// The zot driver will deliver everything destined for a single hub once contact is made (*if* contact is made).
-			// Other drivers will have to do something different here and may need their own query.
-
-			// Note: this requires some tweaking as new posts to long dead hubs once a day will keep them in the
-			// "every 15 minutes" category. We probably need to prioritise them when inserted into the queue
-			// or just prior to this query based on recent and long-term delivery history. If we have good reason to believe
-			// the site is permanently down, there's no reason to attempt delivery at all, or at most not more than once
-			// or twice a day.
-
-			$sqlrandfunc = db_getfunc('rand');
-
-			$r = q("SELECT *,$sqlrandfunc as rn FROM outq WHERE outq_delivered = 0 and outq_scheduled < %s order by rn limit 1",
+			$qItems = q("SELECT outq_hash FROM outq WHERE outq_scheduled < %s ",
 				db_utcnow()
 			);
-			while ($r) {
-				foreach ($r as $rv) {
-					LibQueue::deliver($rv);
-				}
-				$r = q("SELECT *,$sqlrandfunc as rn FROM outq WHERE outq_delivered = 0 and outq_scheduled < %s order by rn limit 1",
-					db_utcnow()
-				);
-			}
-		}
-		if (!$r)
-			return;
 
-		foreach ($r as $rv) {
-			LibQueue::deliver($rv);
-		}
+			if ($qItems) {
+				foreach ($qItems as $qItem) {
+					$deliveries[] = $qItem['outq_hash'];
+				}
+				do_delivery($deliveries, true);
+			}
+		 }
 	}
+
 }

Zotlabs/Daemon/Thumbnail.php

@@ -73,5 +73,7 @@ class Thumbnail {
 				|| (filectime(dbunescbin($attach['content']) . 'thumb') < (time() - 60)))) {
 			$default_controller->Thumb($attach, $preview_style, $preview_width, $preview_height);
 		}
+
+		return;
 	}
 }

Zotlabs/Daemon/Zotconvo.php

@@ -0,0 +1,32 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Lib\Libzot;
+
+class Zotconvo {
+
+	static public function run($argc, $argv) {
+
+		logger('Zotconvo invoked: ' . print_r($argv, true));
+
+		if ($argc != 3) {
+			return;
+		}
+
+		$mid = $argv[2];
+		if (!$mid) {
+			return;
+		}
+
+		$channel = channelx_by_n(intval($argv[1]));
+		if (!$channel) {
+			return;
+		}
+
+		Libzot::fetch_conversation($channel, $mid);
+
+		return;
+
+	}
+}

Zotlabs/Lib/Activity.php

@@ -289,7 +289,7 @@ class Activity {
 
 	static function encode_item_collection($items, $id, $type, $total = 0) {
 
-		if ($total > 30) {
+		if ($total > App::$pager['itemspage']) {
 			$ret = [
 				'id'   => z_root() . '/' . $id,
 				'type' => $type . 'Page',
@@ -419,10 +419,6 @@ class Activity {
 			$objtype = self::activity_obj_mapper($i['obj_type']);
 		}
 
-		if (isset($i['obj']) && $i['obj']) {
-			$ret = Activity::encode_object($i['obj']);
-		}
-
 		if (intval($i['item_deleted'])) {
 			$ret['type']       = 'Tombstone';
 			$ret['formerType'] = $objtype;
@@ -545,6 +541,7 @@ class Activity {
 
 		call_hooks('encode_item', $hookinfo);
 
+
 		return $hookinfo['encoded'];
 
 	}
@@ -625,6 +622,10 @@ class Activity {
 			$atts = ((is_array($item['attach'])) ? $item['attach'] : json_decode($item['attach'], true));
 			if ($atts) {
 				foreach ($atts as $att) {
+					if (!isset($att['type'], $att['href'])) {
+						continue;
+					}
+
 					if (isset($att['type']) && strpos($att['type'], 'image')) {
 						$ret[] = ['type' => 'Image', 'url' => $att['href']];
 					}
@@ -671,6 +672,7 @@ class Activity {
 				}
 			}
 		}
+
 		return $ret;
 	}
 
@@ -678,7 +680,7 @@ class Activity {
 
 		$ret = [];
 
-		if (isset($item['attachment'])) {
+		if (isset($item['attachment']) && is_array($item['attachment'])) {
 			$ptr = $item['attachment'];
 			if (!array_key_exists(0, $ptr)) {
 				$ptr = [$ptr];
@@ -698,6 +700,13 @@ class Activity {
 				if (array_key_exists('name', $att) && $att['name']) {
 					$entry['name'] = html2plain(purify_html($att['name']), 256);
 				}
+				// Friendica attachments don't match the URL in the body.
+				// This makes it more difficult to detect image duplication in bb_attach()
+				// which adds images to plaintext microblog software. For these we need to examine both the
+				// url and image properties.
+				if (isset($att['image']) && is_string($att['image']) && isset($att['url']) && $att['image'] !== $att['url']) {
+					$entry['image'] = $att['image'];
+				}
 				if ($entry) {
 					$ret[] = $entry;
 				}
@@ -820,9 +829,14 @@ class Activity {
 			];
 		}
 
-		$ret['published'] = ((isset($i['created'])) ? datetime_convert('UTC', 'UTC', $i['created'], ATOM_TIME) : datetime_convert());
-		if (isset($i['created'], $i['edited']) && $i['created'] !== $i['edited'])
+		$ret['published'] = datetime_convert('UTC', 'UTC', $i['created'], ATOM_TIME);
+
+		if (isset($i['created'], $i['edited']) && $i['created'] !== $i['edited']) {
 			$ret['updated'] = datetime_convert('UTC', 'UTC', $i['edited'], ATOM_TIME);
+			if ($ret['type'] === 'Create') {
+				$ret['type'] = 'Update';
+			}
+		}
 
 		if (isset($i['app']) && $i['app']) {
 			$ret['generator'] = ['type' => 'Application', 'name' => $i['app']];
@@ -1323,7 +1337,8 @@ class Activity {
 			}
 		}
 
-		$x           = PermissionRoles::role_perms('personal');
+		$role = get_pconfig($channel['channel_id'], 'system', 'permissions_role', 'personal');
+		$x = PermissionRoles::role_perms($role);
 		$their_perms = Permissions::FilledPerms($x['perms_connect']);
 
 		if ($contact && $contact['abook_id']) {
@@ -1348,10 +1363,15 @@ class Activity {
 					return;
 
 				case 'Accept':
+					// They accepted our Follow request.
+					// Set default permissions except for send_stream and post_wall
 
-					// They accepted our Follow request - set default permissions
-
-					set_abconfig($channel['channel_id'], $contact['abook_xchan'], 'system', 'their_perms', $their_perms);
+					foreach ($their_perms as $k => $v) {
+						if(in_array($k, ['send_stream', 'post_wall'])) {
+							$v = 0; // Those will be set once we accept their follow request
+						}
+						set_abconfig($channel['channel_id'], $contact['abook_xchan'], 'their_perms', $k, $v);
+					}
 
 					$abook_instance = $contact['abook_instance'];
 
@@ -1684,6 +1704,8 @@ class Activity {
 			}
 		}
 
+		$group_actor = ($person_obj['type'] === 'Group');
+
 		$r = q("select * from xchan join hubloc on xchan_hash = hubloc_hash where xchan_hash = '%s'",
 			dbesc($url)
 		);
@@ -1702,11 +1724,12 @@ class Activity {
 			);
 
 			// update existing xchan record
-			q("update xchan set xchan_name = '%s', xchan_pubkey = '%s', xchan_addr = '%s', xchan_network = 'activitypub', xchan_name_date = '%s' where xchan_hash = '%s'",
+			q("update xchan set xchan_name = '%s', xchan_pubkey = '%s', xchan_addr = '%s', xchan_network = 'activitypub', xchan_name_date = '%s', xchan_pubforum = %d where xchan_hash = '%s'",
 				dbesc(escape_tags($name)),
 				dbesc(escape_tags($pubkey)),
 				dbesc(escape_tags($webfinger_addr)),
 				dbescdate(datetime_convert()),
+				intval($group_actor),
 				dbesc($url)
 			);
 
@@ -1733,7 +1756,8 @@ class Activity {
 					'xchan_url'       => $profile,
 					'xchan_name'      => escape_tags($name),
 					'xchan_name_date' => datetime_convert(),
-					'xchan_network'   => 'activitypub'
+					'xchan_network'   => 'activitypub',
+					'xchan_pubforum'  => intval($group_actor)
 				]
 			);
 
@@ -1762,6 +1786,10 @@ class Activity {
 				dbesc($url)
 			);
 			if (!$zx) {
+				// FIXME: we might need to fetch and store this url immediately
+				// otherwise at least the first post of a yet unknown author might
+				// be stored with the activitypub url instead of the portable id.
+				// Another solution could be to fix the items after Gprobe has done its work.
 				Master::Summon(['Gprobe', bin2hex($url)]);
 			}
 		}
@@ -2075,16 +2103,29 @@ class Activity {
 
 	}
 
-	static function update_poll($item, $post) {
+	static function update_poll($item_id, $post) {
 
 		$multi   = false;
 		$mid     = $post['mid'];
 		$content = $post['title'];
 
-		if (!$item) {
+		if (!$item_id) {
 			return false;
 		}
 
+		dbq("START TRANSACTION");
+
+		$item = q("SELECT * FROM item WHERE id = %d FOR UPDATE",
+			intval($item_id)
+		);
+
+		if (!$item) {
+			dbq("COMMIT");
+			return false;
+		}
+
+		$item = $item[0];
+
 		$o = json_decode($item['obj'], true);
 		if ($o && array_key_exists('anyOf', $o)) {
 			$multi = true;
@@ -2156,36 +2197,45 @@ class Activity {
 		}
 		logger('updated_poll: ' . print_r($o, true), LOGGER_DATA);
 		if ($answer_found && !$found) {
-			q("update item set obj = '%s', edited = '%s' where id = %d",
+			$u = q("update item set obj = '%s', edited = '%s' where id = %d",
 				dbesc(json_encode($o)),
 				dbesc(datetime_convert()),
 				intval($item['id'])
 			);
 
-			Master::Summon(['Notifier', 'wall-new', $item['id'], $post['mid'] /* trick queueworker de-duplication  */ ]);
-			return true;
+			if ($u) {
+				dbq("COMMIT");
+
+				if ($multi) {
+					// wait some seconds for possible multiple answers to be processed
+					// before calling the notifier
+					sleep(3);
+				}
+
+				Master::Summon(['Notifier', 'wall-new', $item['id']]);
+				return true;
+			}
+
+			dbq("ROLLBACK");
+
 		}
 
+		dbq("COMMIT");
 		return false;
 	}
 
 	static function decode_note($act) {
-
 		$response_activity = false;
-
 		$s = [];
 
 		// These activities should have been handled separately in the Inbox module and should not be turned into posts
 
 		if (
 			in_array($act->type, ['Follow', 'Accept', 'Reject', 'Create', 'Update']) &&
-			is_array($act->obj) &&
-			array_key_exists('type', $act->obj) &&
-			($act->obj['type'] === 'Follow' || ActivityStreams::is_an_actor($act->obj['type']))
+			($act->objprop('type') === 'Follow' || ActivityStreams::is_an_actor($act->objprop('type')))
 		) {
 			return false;
 		}
-
 		// Within our family of projects, Follow/Unfollow of a thread is an internal activity which should not be transmitted,
 		// hence if we receive it - ignore or reject it.
 		// Unfollow is not defined by ActivityStreams, which prefers Undo->Follow.
@@ -2212,7 +2262,7 @@ class Activity {
 			$content = self::get_content($act->obj);
 		}
 
-		$s['mid'] = ((is_array($act->obj) && isset($act->obj['id'])) ? $act->obj['id'] : $act->obj);
+		$s['mid'] = $act->objprop('id') ?: $act->obj;
 
 		if (!$s['mid']) {
 			return false;
@@ -2220,7 +2270,7 @@ class Activity {
 
 		// Friendica sends the diaspora guid in a nonstandard field via AP
 		// If no uuid is provided we will create an uuid v5 from the mid
-		$s['uuid'] = ((is_array($act->obj) && isset($act->obj['diaspora:guid'])) ? $act->obj['diaspora:guid'] : uuid_from_url($s['mid']));
+		$s['uuid'] = (($act->objprop('diaspora:guid')) ?: uuid_from_url($s['mid']));
 
 		$s['parent_mid'] = $act->parent_id;
 
@@ -2228,24 +2278,24 @@ class Activity {
 			$s['created']   = datetime_convert('UTC', 'UTC', $act->data['published']);
 			$s['commented'] = $s['created'];
 		}
-		elseif (is_array($act->obj) && array_key_exists('published', $act->obj)) {
+		elseif ($act->objprop('published')) {
 			$s['created']   = datetime_convert('UTC', 'UTC', $act->obj['published']);
 			$s['commented'] = $s['created'];
 		}
 		if (array_key_exists('updated', $act->data)) {
 			$s['edited'] = datetime_convert('UTC', 'UTC', $act->data['updated']);
 		}
-		elseif (is_array($act->obj) && array_key_exists('updated', $act->obj)) {
+		elseif ($act->objprop('updated')) {
 			$s['edited'] = datetime_convert('UTC', 'UTC', $act->obj['updated']);
 		}
 		if (array_key_exists('expires', $act->data)) {
 			$s['expires'] = datetime_convert('UTC', 'UTC', $act->data['expires']);
 		}
-		elseif (is_array($act->obj) && array_key_exists('expires', $act->obj)) {
+		elseif ($act->objprop('expires')) {
 			$s['expires'] = datetime_convert('UTC', 'UTC', $act->obj['expires']);
 		}
 
-		if ($act->type === 'Invite' && is_array($act->obj) && array_key_exists('type', $act->obj) && $act->obj['type'] === 'Event') {
+		if ($act->type === 'Invite' && $act->objprop('type') === 'Event') {
 			$s['mid'] = $s['parent_mid'] = $act->id;
 		}
 
@@ -2254,9 +2304,18 @@ class Activity {
 			$response_activity = true;
 
 			$s['mid'] = $act->id;
-			$s['uuid'] = ((is_array($act->data) && isset($act->data['diaspora:guid'])) ? $act->data['diaspora:guid'] : uuid_from_url($s['mid']));
+			$s['uuid'] = ((!empty($act->data['diaspora:guid'])) ? $act->data['diaspora:guid'] : uuid_from_url($s['mid']));
 
-			$s['parent_mid'] = ((is_array($act->obj) && isset($act->obj['id'])) ? $act->obj['id'] : $act->obj);
+			if ($act->objprop('inReplyTo')) {
+				$s['parent_mid'] = $act->objprop('inReplyTo');
+			}
+
+			$s['thr_parent'] = $act->objprop('id') ?: $act->obj;
+
+			if (empty($s['parent_mid']) || empty($s['thr_parent'])) {
+				logger('response activity without parent_mid or thr_parent');
+				return;
+			}
 
 			// over-ride the object timestamp with the activity
 
@@ -2268,7 +2327,7 @@ class Activity {
 				$s['edited'] = datetime_convert('UTC', 'UTC', $act->data['updated']);
 			}
 
-			$obj_actor = ((isset($act->obj['actor'])) ? $act->obj['actor'] : $act->get_actor('attributedTo', $act->obj));
+			$obj_actor = $act->objprop('actor') ?: $act->get_actor('attributedTo', $act->obj);
 
 			if (!isset($obj_actor['id'])) {
 				return false;
@@ -2287,7 +2346,7 @@ class Activity {
 			}
 
 			// handle event RSVPs
-			if (($act->obj['type'] === 'Event') || ($act->obj['type'] === 'Invite' && array_path_exists('object/type', $act->obj) && $act->obj['object']['type'] === 'Event')) {
+			if (($act->objprop('type') === 'Event') || ($act->objprop('type') === 'Invite' && array_path_exists('object/type', $act->obj) && $act->obj['object']['type'] === 'Event')) {
 				if ($act->type === 'Accept') {
 					$content['content'] = sprintf(t('Will attend %s\'s event'), $mention) . EOL . EOL . $content['content'];
 				}
@@ -2322,7 +2381,7 @@ class Activity {
 			$s['item_thread_top'] = 1;
 
 			// it is a parent node - decode the comment policy info if present
-			if (isset($act->obj['commentPolicy'])) {
+			if ($act->objprop('commentPolicy')) {
 				$until = strpos($act->obj['commentPolicy'], 'until=');
 				if ($until !== false) {
 					$s['comments_closed'] = datetime_convert('UTC', 'UTC', substr($act->obj['commentPolicy'], $until + 6));
@@ -2348,7 +2407,7 @@ class Activity {
 		$s['summary'] = self::bb_content($content, 'summary');
 		$s['body']    = ((self::bb_content($content, 'bbcode') && (!$response_activity)) ? self::bb_content($content, 'bbcode') : self::bb_content($content, 'content'));
 
-		if (isset($act->obj['quoteUrl'])) {
+		if ($act->objprop('quoteUrl')) {
 			$quote_bbcode = self::get_quote_bbcode($act->obj['quoteUrl']);
 
 			if ($s['body']) {
@@ -2361,21 +2420,24 @@ class Activity {
 		$s['verb'] = self::activity_decode_mapper($act->type);
 
 		// Mastodon does not provide update timestamps when updating poll tallies which means race conditions may occur here.
-		if ($act->type === 'Update' && $act->obj['type'] === 'Question' && $s['edited'] === $s['created']) {
+		if ($act->type === 'Update' && $act->objprop('type') === 'Question' && $s['edited'] === $s['created']) {
 			$s['edited'] = datetime_convert();
 		}
 
-		if (in_array($act->type, ['Delete', 'Undo', 'Tombstone']) || ($act->type === 'Create' && $act->obj['type'] === 'Tombstone')) {
+		if (in_array($act->type, ['Delete', 'Undo', 'Tombstone']) || ($act->type === 'Create' && (isset($act->obj['type']) && $act->obj['type'] === 'Tombstone'))) {
 			$s['item_deleted'] = 1;
 		}
 
-		$s['obj_type'] = self::activity_obj_decode_mapper($act->obj['type']);
+		if (isset($act->obj['type'])) {
+			$s['obj_type'] = self::activity_obj_decode_mapper($act->obj['type']);
+		}
+
 		if ($s['obj_type'] === ACTIVITY_OBJ_NOTE && $s['mid'] !== $s['parent_mid']) {
 			$s['obj_type'] = ACTIVITY_OBJ_COMMENT;
 		}
 
 		$s['obj'] = $act->obj;
-		if (is_array($s['obj']) && array_path_exists('actor/id', $s['obj'])) {
+		if (array_path_exists('actor/id', $s['obj'])) {
 			$s['obj']['actor'] = $s['obj']['actor']['id'];
 		}
 
@@ -2451,29 +2513,27 @@ class Activity {
 				$s['attach'] = $a;
 			}
 
-			$a = self::decode_iconfig($act->obj);
+			$a = self::decode_iconfig($act->data);
 			if ($a) {
 				$s['iconfig'] = $a;
 			}
 		}
 
-		if (array_key_exists('type', $act->obj)) {
 
-			// Objects that might have media attachments which aren't already provided in the content element.
-			// We'll check specific media objects separately.
+		// Objects that might have media attachments which aren't already provided in the content element.
+		// We'll check specific media objects separately.
 
-			if (in_array($act->obj['type'], ['Article', 'Document', 'Event', 'Note', 'Page', 'Place', 'Question']) && isset($s['attach']) && $s['attach']) {
-				$s = self::bb_attach($s);
-			}
+		if (in_array($act->objprop('type',''), ['Article', 'Document', 'Event', 'Note', 'Page', 'Place', 'Question']) && !empty($s['attach'])) {
+			$s = self::bb_attach($s);
+		}
 
-			if ($act->obj['type'] === 'Question' && in_array($act->type, ['Create', 'Update'])) {
-				if (array_key_exists('endTime', $act->obj)) {
-					$s['comments_closed'] = datetime_convert('UTC', 'UTC', $act->obj['endTime']);
-				}
+		if ($act->objprop('type') === 'Question' && in_array($act->type, ['Create', 'Update'])) {
+			if ($act->objprop('endTime')) {
+				$s['comments_closed'] = datetime_convert('UTC', 'UTC', $act->obj['endTime']);
 			}
 		}
 
-		if (array_key_exists('closed', $act->obj)) {
+		if ($act->objprop('closed')) {
 			$s['comments_closed'] = datetime_convert('UTC', 'UTC', $act->obj['closed']);
 		}
 
@@ -2481,14 +2541,18 @@ class Activity {
 			if ($act->type === 'Announce') {
 				$s['author_xchan'] = self::get_attributed_to_actor_url($act);
 				$s['mid'] = $act->obj['id'];
-				$s['parent_mid'] = $act->obj['id'];
+
+				// Do not force new thread if the announce is from a group actor
+				if ($act->actor['type'] !== 'Group') {
+					$s['parent_mid'] = $act->obj['id'];
+				}
 			}
 
 			// we will need a hook here to extract magnet links e.g. peertube
 			// right now just link to the largest mp4 we find that will fit in our
 			// standard content region
 
-			if ($act->obj['type'] === 'Video') {
+			if ($act->objprop('type') === 'Video') {
 
 				$vtypes = [
 					'video/mp4',
@@ -2502,7 +2566,7 @@ class Activity {
 
 				// try to find a poster to display on the video element
 
-				if (array_key_exists('icon',$act->obj)) {
+				if ($act->objprop('icon')) {
 					if (is_array($act->obj['icon'])) {
 						if (array_key_exists(0,$act->obj['icon'])) {
 							$ptr = $act->obj['icon'];
@@ -2523,7 +2587,7 @@ class Activity {
 				$tag = (($poster) ? '[video poster="' . $poster . '"]' : '[video]' );
 				$ptr = null;
 
-				if (array_key_exists('url',$act->obj)) {
+				if ($act->objprop('url')) {
 					if (is_array($act->obj['url'])) {
 						if (array_key_exists(0,$act->obj['url'])) {
 							$ptr = $act->obj['url'];
@@ -2556,20 +2620,20 @@ class Activity {
 					if ($mps) {
 						usort($mps,[ '\Zotlabs\Lib\Activity', 'vid_sort' ]);
 						foreach ($mps as $m) {
-							if (intval($m['height']) < 500 && Activity::media_not_in_body($m['href'],$s['body'])) {
+							if (intval($m['height']) < 500 && self::media_not_in_body($m['href'],$s['body'])) {
 								$s['body'] = $tag . $m['href'] . '[/video]' . "\r\n" . $s['body'];
 								break;
 							}
 						}
 					}
-					elseif (is_string($act->obj['url']) && Activity::media_not_in_body($act->obj['url'],$s['body'])) {
+					elseif (is_string($act->obj['url']) && self::media_not_in_body($act->obj['url'],$s['body'])) {
 						$s['body'] = $tag . $act->obj['url'] . '[/video]' . "\r\n" . $s['body'];
 					}
 
 				}
 			}
 
-			if ($act->obj['type'] === 'Audio') {
+			if ($act->objprop('type') === 'Audio') {
 
 				$atypes = [
 					'audio/mpeg',
@@ -2601,7 +2665,7 @@ class Activity {
 
 			}
 
-			if ($act->obj['type'] === 'Image' && strpos($s['body'], 'zrl=') === false) {
+			if ($act->objprop('type') === 'Image' && strpos($s['body'], 'zrl=') === false) {
 
 				$ptr = null;
 
@@ -2630,7 +2694,7 @@ class Activity {
 			}
 
 
-			if ($act->obj['type'] === 'Page' && !$s['body']) {
+			if ($act->objprop('type') === 'Page' && !$s['body']) {
 
 				$ptr  = null;
 				$purl = EMPTY_STR;
@@ -2670,7 +2734,7 @@ class Activity {
 			}
 		}
 
-		if (in_array($act->obj['type'], ['Note', 'Article', 'Page'])) {
+		if (in_array($act->objprop('type', ''), ['Note', 'Article', 'Page'])) {
 			$ptr = null;
 
 			if (array_key_exists('url', $act->obj)) {
@@ -2705,10 +2769,8 @@ class Activity {
 			$s['item_private'] = 0;
 		}
 
-		if (is_array($act->obj)) {
-			if (array_key_exists('directMessage', $act->obj) && intval($act->obj['directMessage'])) {
-				$s['item_private'] = 2;
-			}
+		if ($act->objprop('directMessage')) {
+			$s['item_private'] = 2;
 		}
 
 		$ap_rawmsg = '';
@@ -2781,8 +2843,9 @@ class Activity {
 			set_iconfig($s, 'diaspora', 'fields', $diaspora_rawmsg, 1);
 		}
 
-		set_iconfig($s, 'activitypub', 'recips', $act->raw_recips);
-
+		if ($act->raw_recips) {
+			set_iconfig($s, 'activitypub', 'recips', $act->raw_recips);
+		}
 
 		$hookinfo = [
 			'act' => $act,
@@ -2790,7 +2853,6 @@ class Activity {
 		];
 
 		call_hooks('decode_note', $hookinfo);
-
 		return $hookinfo['s'];
 
 	}
@@ -2798,6 +2860,7 @@ class Activity {
 	static function store($channel, $observer_hash, $act, $item, $fetch_parents = true, $force = false) {
 		$is_sys_channel = is_sys_channel($channel['channel_id']);
 		$is_child_node  = false;
+		$parent = null;
 
 		// TODO: not implemented
 		// Pleroma scrobbles can be really noisy and contain lots of duplicate activities. Disable them by default.
@@ -2809,26 +2872,31 @@ class Activity {
 			$is_child_node = true;
 		}
 
-		$allowed = false;
+		if (empty($item['item_fetched'])) {
+			$item['owner_xchan'] = $observer_hash;
+		}
 
-		// TODO: not implemented
-		// $permit_mentions = intval(PConfig::Get($channel['channel_id'], 'system','permit_all_mentions') && i_am_mentioned($channel,$item));
+		$allowed = false;
+		$permit_mentions = intval(PConfig::Get($channel['channel_id'], 'system','permit_all_mentions') && i_am_mentioned($channel, $item));
 
 		if ($is_child_node) {
 
-			$p = q("select * from item where mid = '%s' and uid = %d and item_wall = 1",
+			$parent = q("select * from item where mid = '%s' and uid = %d",
 				dbesc($item['parent_mid']),
 				intval($channel['channel_id'])
 			);
-			if ($p) {
+
+			// TODO: if we do not have a parent stop here and move the fetch to background?
+
+			if ($parent && $parent[0]['item_wall']) {
 				// set the owner to the owner of the parent
-				$item['owner_xchan'] = $p[0]['owner_xchan'];
+				$item['owner_xchan'] = $parent[0]['owner_xchan'];
 
 				// quietly reject group comment boosts by group owner
 				// (usually only sent via ActivityPub so groups will work on microblog platforms)
 				// This catches those activities if they slipped in via a conversation fetch
 
-				if ($p[0]['parent_mid'] !== $item['parent_mid']) {
+				if ($parent[0]['parent_mid'] !== $item['parent_mid']) {
 					if ($item['verb'] === 'Announce' && $item['author_xchan'] === $item['owner_xchan']) {
 						logger('group boost activity by group owner rejected');
 						return;
@@ -2837,8 +2905,8 @@ class Activity {
 
 				// check permissions against the author, not the sender
 				$allowed = perm_is_allowed($channel['channel_id'], $item['author_xchan'], 'post_comments');
-				if ((!$allowed)/* && $permit_mentions*/) {
-					if ($p[0]['owner_xchan'] === $channel['channel_hash']) {
+				if ((!$allowed) && $permit_mentions) {
+					if ($parent[0]['owner_xchan'] === $channel['channel_hash']) {
 						$allowed = false;
 					}
 					else {
@@ -2847,7 +2915,7 @@ class Activity {
 				}
 
 				// TODO: not implemented
-				/*if (absolutely_no_comments($p[0])) {
+				/*if (absolutely_no_comments($parent[0])) {
 					$allowed = false;
 				}*/
 
@@ -2869,13 +2937,14 @@ class Activity {
 			else {
 
 				$allowed = true;
+
 				// reject public stream comments that weren't sent by the conversation owner
 				if ($is_sys_channel && $item['owner_xchan'] !== $observer_hash && !$fetch_parents) {
 					$allowed = false;
 				}
 			}
 
-			if ($p && $p[0]['obj_type'] === 'Question') {
+			if ($parent && $parent[0]['obj_type'] === 'Question') {
 				if ($item['obj_type'] === ACTIVITY_OBJ_COMMENT && $item['title'] && (!$item['body'])) {
 					$item['obj_type'] = 'Answer';
 				}
@@ -2885,13 +2954,13 @@ class Activity {
 
 			// The $item['item_fetched'] flag is set in fetch_and_store_parents().
 			// In this case we should check against author permissions because sender is not owner.
-			if (perm_is_allowed($channel['channel_id'], ((isset($item['item_fetched']) && $item['item_fetched']) ? $item['author_xchan'] : $observer_hash), 'send_stream') || $is_sys_channel) {
+			if (perm_is_allowed($channel['channel_id'], ((!empty($item['item_fetched'])) ? $item['author_xchan'] : $observer_hash), 'send_stream') || $is_sys_channel) {
 				$allowed = true;
 			}
-			// TODO: not implemented
-			/*if ($permit_mentions) {
+
+			if ($permit_mentions) {
 				$allowed = true;
-			}*/
+			}
 		}
 
 		if (tgroup_check($channel['channel_id'], $item) && (!$is_child_node)) {
@@ -3004,34 +3073,25 @@ class Activity {
 			$item['item_verified'] = 1;
 		}
 
-		$parent = null;
-
 		if ($is_child_node) {
-
-			$parent = q("select * from item where mid = '%s' and uid = %d limit 1",
-				dbesc($item['parent_mid']),
-				intval($item['uid'])
-			);
-
 			if (!$parent) {
 				if (!plugin_is_installed('pubcrawl')) {
 					return;
 				}
-				else {
-					$fetch = false;
 
-					// TODO: debug
-					// if (perm_is_allowed($channel['channel_id'],$observer_hash,'send_stream') && (PConfig::Get($channel['channel_id'],'system','hyperdrive',true) || $act->type === 'Announce')) {
-					if (perm_is_allowed($channel['channel_id'], $observer_hash, 'send_stream') || $is_sys_channel) {
-						$fetch = (($fetch_parents) ? self::fetch_and_store_parents($channel, $observer_hash, $item, $force) : false);
-					}
+				$fetch = false;
 
-					if ($fetch) {
-						$parent = q("select * from item where mid = '%s' and uid = %d limit 1",
-							dbesc($item['parent_mid']),
-							intval($item['uid'])
-						);
-					}
+				// TODO: debug
+				// if (perm_is_allowed($channel['channel_id'],$observer_hash,'send_stream') && (PConfig::Get($channel['channel_id'],'system','hyperdrive',true) || $act->type === 'Announce')) {
+				if (perm_is_allowed($channel['channel_id'], $observer_hash, 'send_stream') || $is_sys_channel) {
+					$fetch = (($fetch_parents) ? self::fetch_and_store_parents($channel, $observer_hash, $item, $force) : false);
+				}
+
+				if ($fetch) {
+					$parent = q("select * from item where mid = '%s' and uid = %d",
+						dbesc($item['parent_mid']),
+						intval($item['uid'])
+					);
 				}
 			}
 
@@ -3040,6 +3100,7 @@ class Activity {
 				return;
 			}
 
+/*
 			if ($parent[0]['parent_mid'] !== $item['parent_mid']) {
 				$item['thr_parent'] = $item['parent_mid'];
 			}
@@ -3047,6 +3108,7 @@ class Activity {
 				$item['thr_parent'] = $parent[0]['parent_mid'];
 			}
 			$item['parent_mid'] = $parent[0]['parent_mid'];
+*/
 
 			/*
 			 *
@@ -3194,7 +3256,7 @@ class Activity {
 			$item = $hookinfo['item'];
 
 			if ($item) {
-				$item['item_fetched'] = 1;
+				$item['item_fetched'] = true;
 
 				if (intval($channel['channel_system']) && intval($item['item_private'])) {
 					$p = [];
@@ -3308,7 +3370,7 @@ class Activity {
 
 			return false;
 		}
-	*/
+
 	static public function fetch_and_store_replies($channel, $arr) {
 
 		logger('fetching replies');
@@ -3365,6 +3427,7 @@ class Activity {
 		}
 
 	}
+*/
 
 /* this is deprecated and not used anymore
 	static function announce_note($channel, $observer_hash, $act) {
@@ -3640,7 +3703,21 @@ class Activity {
 				if ($a['type'] === 'image/svg+xml' && strpos($item['body'], '[/svg]')) {
 					continue;
 				}
-				if (self::media_not_in_body($a['href'], $item['body'])) {
+				// Friendica attachment weirdness
+				// Check both the attachment image and href since they can be different and the one in the href is a different link with different resolution.
+				// Otheriwse you'll get duplicated images
+				if (isset($a['image'])) {
+					if (self::media_not_in_body($a['image'], $item['body']) && self::media_not_in_body($a['href'], $item['body'])) {
+						if (isset($a['name']) && $a['name']) {
+							$alt = htmlspecialchars($a['name'], ENT_QUOTES);
+							$item['body'] = '[img=' . $a['href']  . ']' . $alt . '[/img]' . "\r\n" . $item['body'];
+						} else {
+							$item['body'] = '[img]' . $a['href'] . '[/img]' . "\r\n" . $item['body'];
+						}
+					}
+					continue;
+				}
+				elseif (self::media_not_in_body($a['href'], $item['body'])) {
 					if (isset($a['name']) && $a['name']) {
 						$alt = htmlspecialchars($a['name'], ENT_QUOTES);
 						$item['body'] = '[img=' . $a['href']  . ']' . $alt . '[/img]' . "\r\n" . $item['body'];
@@ -3920,18 +3997,18 @@ class Activity {
 
 		switch ($options) {
 			case 'activitypub':
-				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where hubloc_hash = '%s' and hubloc_deleted = 0 ",
+				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where hubloc_hash = '%s' and hubloc_deleted = 0 order by hubloc_id desc",
 					dbesc($url)
 				);
 				break;
 			case 'zot6':
-				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where hubloc_id_url = '%s' and hubloc_deleted = 0 ",
+				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where hubloc_id_url = '%s' and hubloc_deleted = 0 order by hubloc_id desc",
 					dbesc($url)
 				);
 				break;
 			case 'all':
 			default:
-				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where ( hubloc_id_url = '%s' OR hubloc_hash = '%s' ) and hubloc_deleted = 0 ",
+				$hublocs = q("select * from hubloc left join xchan on hubloc_hash = xchan_hash where ( hubloc_id_url = '%s' OR hubloc_hash = '%s' ) and hubloc_deleted = 0 order by hubloc_id desc",
 					dbesc($url),
 					dbesc($url)
 				);
@@ -3991,9 +4068,9 @@ class Activity {
 			if ($act->is_valid()) {
 				$content = self::get_content($act->obj);
 
-				$ret .= "[share author='" . urlencode($act->actor['name']) .
+				$ret .= "[share author='" . urlencode($act->actor['name'] ?? $act->actor['preferredUsername']) .
 					"' profile='" . $act->actor['id'] .
-					"' avatar='" . $act->actor['icon']['url'] .
+					"' avatar='" . ($act->actor['icon']['url'] ?? z_root() . '/' . get_default_profile_photo(80)) .
 					"' link='" . $act->obj['id'] .
 					"' auth='" . ((is_matrix_url($act->actor['id'])) ? 'true' : 'false') .
 					"' posted='" . $act->obj['published'] .

Zotlabs/Lib/ActivityStreams.php

@@ -113,7 +113,7 @@ class ActivityStreams {
 			// fetch recursive or embedded activities
 
 			if ($this->obj && is_array($this->obj) && array_key_exists('object', $this->obj)) {
-				$this->obj['object'] = $this->get_compound_property($this->obj['object']);
+				$this->obj['object'] = $this->get_compound_property('object', $this->obj);
 			}
 
 			if ($this->obj && is_array($this->obj) && isset($this->obj['actor']))
@@ -126,6 +126,7 @@ class ActivityStreams {
 			if ((!$this->parent_id) && is_array($this->obj) && isset($this->obj['inReplyTo'])) {
 				$this->parent_id = $this->obj['inReplyTo'];
 			}
+
 			if ((!$this->parent_id) && is_array($this->obj) && isset($this->obj['id'])) {
 				$this->parent_id = $this->obj['id'];
 			}
@@ -145,36 +146,59 @@ class ActivityStreams {
 		$this->saved_recips = $arr;
 	}
 
+	/**
+	 * @brief get single property from Activity object
+	 *
+	 * @param string $property
+	 * @param mixed $default return value if property or object not set
+	 *    or object is a string id which could not be fetched.
+	 * @return mixed
+	 */
+	public function objprop(string $property, mixed $default = false): mixed {
+		$x = $this->get_property_obj($property, $this->obj);
+		return (isset($x)) ? $x : $default;
+	}
+
 	/**
 	 * @brief Collects all recipients.
 	 *
-	 * @param string $base
+	 * @param mixed $base
 	 * @param string $namespace (optional) default empty
 	 * @return array
 	 */
-	function collect_recips($base = '', $namespace = '') {
-		$x = [];
+	public function collect_recips(mixed $base = '', string $namespace = ''): array {
+		$result = [];
+		$tmp = [];
 
 		$fields = ['to', 'cc', 'bto', 'bcc', 'audience'];
-		foreach ($fields as $f) {
-			$y = $this->get_compound_property($f, $base, $namespace);
-			if ($y) {
-				if (!is_array($this->raw_recips)) {
-					$this->raw_recips = [];
-				}
-
-				if (!is_array($y)) {
-					$y = [$y];
-				}
-				$this->raw_recips[$f] = $y;
-				$x                    = array_merge($x, $y);
+		foreach ($fields as $field) {
+			// don't expand these yet
+			$values = $this->get_property_obj($field, $base, $namespace);
+			if ($values) {
+				$values = force_array($values);
+				$tmp[$field] = $values;
+				$result = array_values(array_unique(array_merge($result, $values)));
+			}
+			// Merge the object recipients if they exist.
+			$values = $this->objprop($field);
+			if ($values) {
+				$values = force_array($values);
+				$tmp[$field] = ((isset($tmp[$field])) ? array_merge($tmp[$field], $values) : $values);
+				$result = array_values(array_unique(array_merge($result, $values)));
+			}
+			// remove duplicates
+			if (isset($tmp[$field])) {
+				$tmp[$field] = array_values(array_unique($tmp[$field]));
 			}
 		}
-// not yet ready for prime time
-//		$x = $this->expand($x,$base,$namespace);
-		return $x;
+		$this->raw_recips = $tmp;
+
+		// not yet ready for prime time
+		//      $result = $this->expand($result,$base,$namespace);
+		return $result;
 	}
 
+
 	function expand($arr, $base = '', $namespace = '') {
 		$ret = [];
 
@@ -341,6 +365,7 @@ class ActivityStreams {
 	 */
 	function get_compound_property($property, $base = '', $namespace = '', $first = false) {
 		$x = $this->get_property_obj($property, $base, $namespace);
+
 		if ($this->is_url($x)) {
 			$y = $this->fetch_property($x);
 			if (is_array($y)) {

Zotlabs/Lib/Connect.php

@@ -69,7 +69,8 @@ class Connect {
 		$xchan_hash = '';
 		$sql_options = (($protocol) ? " and xchan_network = '" . dbesc($protocol) . "' " : '');
 
-		$r = q("SELECT * FROM xchan LEFT JOIN hubloc ON xchan_hash = hubloc_hash WHERE ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s') $sql_options ORDER BY hubloc_id DESC",
+		// We need both, the xchan and the hubloc here hence use JOIN instead of LEFT JOIN
+		$r = q("SELECT * FROM xchan JOIN hubloc ON xchan_hash = hubloc_hash WHERE ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s') $sql_options ORDER BY hubloc_id DESC",
 			dbesc($url),
 			dbesc($url),
 			dbesc($url)
@@ -109,10 +110,12 @@ class Connect {
 
 		if ($wf || $d) {
 
+			$xchan_hash = (($wf) ? $wf : $url);
+
 			// something was discovered - find the record which was just created.
 
 			$r = q("select * from xchan where ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s' ) $sql_options",
-				dbesc($wf ?? $url),
+				dbesc($xchan_hash),
 				dbesc($url),
 				dbesc($url)
 			);

Zotlabs/Lib/DReport.php

@@ -94,19 +94,6 @@ class DReport {
 		if(! $c)
 			return false;
 
-		// legacy zot recipients add a space and their name to the xchan. remove it if true.
-
-		$legacy_recipient = strpos($dr['recipient'], ' ');
-		if($legacy_recipient !== false) {
-			$legacy_recipient_parts = explode(' ', $dr['recipient'], 2);
-			$rxchan = $legacy_recipient_parts[0];
-		}
-		else {
-			$rxchan = $dr['recipient'];
-		}
-
-
-
 		// is the recipient one of our connections, or do we want to store every report?
 
 		$pcf = get_pconfig($c[0]['channel_id'],'system','dreport_store_all');
@@ -117,7 +104,7 @@ class DReport {
 		// So if a remote site says they can't find us, that's no big surprise
 		// and just creates a lot of extra report noise
 
-		if(($dr['location'] !== z_root()) && ($dr['sender'] === $rxchan) && ($dr['status'] === 'recipient not found'))
+		if(($dr['location'] !== z_root()) && ($dr['sender'] === $dr['recipient']) && ($dr['status'] === 'recipient not found'))
 			return false;
 
 		// If you have a private post with a recipient list, every single site is going to report
@@ -126,14 +113,14 @@ class DReport {
 		// have a channel on that site.
 
 		$r = q("select hubloc_id from hubloc where hubloc_hash = '%s' and hubloc_url = '%s'",
-			dbesc($rxchan),
+			dbesc($dr['recipient']),
 			dbesc($dr['location'])
 		);
 		if((! $r) && ($dr['status'] === 'recipient_not_found'))
 			return false;
 
 		$r = q("select abook_id from abook where abook_xchan = '%s' and abook_channel = %d limit 1",
-			dbesc($rxchan),
+			dbesc($dr['recipient']),
 			intval($c[0]['channel_id'])
 		);
 		if($r)

Zotlabs/Lib/Enotify.php

@@ -828,7 +828,7 @@ class Enotify {
 				: (($item['obj_type'] === 'Answer') ? sprintf( t('voted on %s\'s poll'), '[bdi]' . $item['owner']['xchan_name'] . '[/bdi]') : sprintf( t('commented on %s\'s post'), '[bdi]' . $item['owner']['xchan_name'] . '[/bdi]'))
 			);
 
-			if($item['verb'] === ACTIVITY_SHARE) {
+			if($item['verb'] === ACTIVITY_SHARE && empty($item['owner']['xchan_pubforum'])) {
 				$itemem_text = sprintf( t('repeated %s\'s post'), '[bdi]' . $item['author']['xchan_name'] . '[/bdi]');
 			}
 
@@ -860,7 +860,7 @@ class Enotify {
 
 		// convert this logic into a json array just like the system notifications
 
-		$who = (($item['verb'] === ACTIVITY_SHARE) ? 'owner' : 'author');
+		$who = (($item['verb'] === ACTIVITY_SHARE && empty($item['owner']['xchan_pubforum'])) ? 'owner' : 'author');
 		$body = html2plain(bbcode($item['body'], ['drop_media' => true, 'tryoembed' => false]), 75, true);
 		if ($body) {
 			$body = htmlentities($body, ENT_QUOTES, 'UTF-8', false);
@@ -869,7 +869,7 @@ class Enotify {
 		$x = array(
 			'notify_link' => $item['llink'],
 			'name' => $item[$who]['xchan_name'],
-			'addr' => $item[$who]['xchan_addr'] ?? $item[$who]['xchan_url'],
+			'addr' => $item[$who]['xchan_addr'] ? $item[$who]['xchan_addr'] : $item[$who]['xchan_url'],
 			'url' => $item[$who]['xchan_url'],
 			'photo' => $item[$who]['xchan_photo_s'],
 			'when' => (($edit) ? datetime_convert('UTC', date_default_timezone_get(), $item['edited']) : datetime_convert('UTC', date_default_timezone_get(), $item['created'])),

Zotlabs/Lib/Libsync.php

@@ -135,14 +135,13 @@ class Libsync {
 				$info['collection_members'] = $r;
 		}
 
-		$interval = ((get_config('system', 'delivery_interval') !== false)
-			? intval(get_config('system', 'delivery_interval')) : 2);
+		$interval = get_config('queueworker', 'queue_interval', 500000);
 
 		logger('Packet: ' . print_r($info, true), LOGGER_DATA, LOG_DEBUG);
 
 		$total = count($synchubs);
 		foreach ($synchubs as $hub) {
-			$hash = random_string();
+			$hash = new_uuid();
 			$n    = Libzot::build_packet($channel, 'sync', $env_recips, json_encode($info), 'hz', $hub['hubloc_sitekey'], $hub['site_crypto']);
 			Queue::insert([
 				'hash'       => $hash,
@@ -155,19 +154,26 @@ class Libsync {
 			]);
 
 
+			/*
 			$x = q("select count(outq_hash) as total from outq where outq_delivered = 0");
+
 			if (intval($x[0]['total']) > intval(get_config('system', 'force_queue_threshold', 3000))) {
 				logger('immediate delivery deferred.', LOGGER_DEBUG, LOG_INFO);
 				Queue::update($hash);
 				continue;
 			}
-
+			*/
 
 			Master::Summon(['Deliver', $hash]);
-			$total = $total - 1;
 
-			if ($interval && $total)
-				@time_sleep_until(microtime(true) + (float)$interval);
+			/*
+			$total = $total - 1;
+			*/
+
+			if ($interval) {
+				usleep($interval);
+			}
+
 		}
 	}
 
@@ -192,7 +198,10 @@ class Libsync {
 				dbesc($sender)
 			);
 
-			$DR = new DReport(z_root(), $sender, $d, 'sync');
+			$mid = 'sync';
+
+
+			$DR = new DReport(z_root(), $sender, $d, $mid);
 
 			if (!$r) {
 				$DR->update('recipient not found');
@@ -202,7 +211,6 @@ class Libsync {
 
 			$channel = $r[0];
 
-			$mid = 'sync';
 
 			$DR->set_name($channel['channel_name'] . ' <' . channel_reddress($channel) . '>');
 
@@ -297,7 +305,7 @@ class Libsync {
 
 			if (array_key_exists('item', $arr) && $arr['item']) {
 				sync_items($channel, $arr['item'], ((array_key_exists('relocate', $arr)) ? $arr['relocate'] : null));
-				$mid = $arr['item']['mid'] . '#sync';
+				$mid = $arr['item'][0]['message_id'] . '#sync';
 			}
 
 			// deprecated, maintaining for a few months for upward compatibility
@@ -750,7 +758,6 @@ class Libsync {
 
 			$result[] = $DR->get();
 		}
-
 		return $result;
 	}
 
@@ -759,13 +766,17 @@ class Libsync {
 	 *
 	 * @param array $sender
 	 * @param array $arr
-	 * @param boolean $absolute (optional) default false
 	 * @return array
 	 */
 
-	static function sync_locations($sender, $arr, $absolute = false) {
+	static function sync_locations($sender, $arr) {
+
+		$ret = [
+			'change_message' => '',
+			'changed' => false,
+			'message' => ''
+		];
 
-		$ret = [];
 		$what = '';
 		$changed = false;
 
@@ -780,10 +791,7 @@ class Libsync {
 
 		if (isset($arr['locations']) && $arr['locations']) {
 
-			if ($absolute)
-				Libzot::check_location_move($sender['hash'], $arr['locations']);
-
-			$xisting = q("select * from hubloc where hubloc_hash = '%s'",
+			$xisting = q("select * from hubloc where hubloc_hash = '%s' and hubloc_deleted = 0",
 				dbesc($sender['hash'])
 			);
 
@@ -855,7 +863,9 @@ class Libsync {
 						);
 					}
 
-					// update connection timestamp if this is the site we're talking to
+					// Update connection timestamp if this is the site we're talking to.
+					// Also mark all entries from the current site with different sitekeys
+					// deleted (the site has been re-installed)
 					// This only happens when called from import_xchan
 
 					$current_site = false;
@@ -869,6 +879,12 @@ class Libsync {
 							intval($r[0]['hubloc_id']),
 							dbesc($t)
 						);
+
+						q("update hubloc set hubloc_error = 1, hubloc_deleted = 1 where hubloc_url = '%s' and hubloc_sitekey != '%s'",
+							dbesc($r[0]['hubloc_url']),
+							dbesc($r[0]['hubloc_sitekey'])
+						);
+
 						$current_site = true;
 					}
 
@@ -918,14 +934,7 @@ class Libsync {
 						$what    .= 'primary_hub ';
 						$changed = true;
 					}
-					elseif ($absolute) {
-						// Absolute sync - make sure the current primary is correctly reflected in the xchan
-						$pr = hubloc_change_primary($r[0]);
-						if ($pr) {
-							$what    .= 'xchan_primary ';
-							$changed = true;
-						}
-					}
+
 					if (intval($r[0]['hubloc_deleted']) && (!intval($location['deleted']))) {
 						q("update hubloc set hubloc_deleted = 0, hubloc_updated = '%s' where hubloc_id_url = '%s'",
 							dbesc(datetime_convert()),
@@ -994,9 +1003,9 @@ class Libsync {
 				}
 			}
 
-			// get rid of any hubs we have for this channel which weren't reported.
+			// get rid of any hublocs we have for this channel which weren't reported.
 
-			if ($absolute && $xisting) {
+			if ($xisting) {
 				foreach ($xisting as $x) {
 					if (!array_key_exists('updated', $x)) {
 						logger('Deleting unreferenced hub location ' . $x['hubloc_addr']);

Zotlabs/Lib/Libzot.php

@@ -101,7 +101,6 @@ class Libzot {
 	static function build_packet($channel, $type = 'activity', $recipients = null, $msg = [], $encoding = 'activitystreams', $remote_key = null, $methods = '') {
 
 		$sig_method = get_config('system', 'signature_algorithm', 'sha256');
-
 		$data = [
 			'type'     => $type,
 			'encoding' => $encoding,
@@ -115,9 +114,9 @@ class Libzot {
 		}
 
 		if ($msg) {
-			$actor = channel_url($channel);
-			if ($encoding === 'activitystreams' && array_key_exists('actor', $msg) && is_string($msg['actor']) && $actor === $msg['actor']) {
-				$msg = JSalmon::sign($msg, $actor, $channel['channel_prvkey']);
+			$actors = get_hubloc_id_urls_by_x($channel['channel_hash']);
+			if ($encoding === 'activitystreams' && array_key_exists('actor', $msg) && is_string($msg['actor']) && in_array($msg['actor'], $actors)) {
+				$msg = JSalmon::sign($msg, $actors[0], $channel['channel_prvkey']);
 			}
 			$data['data'] = $msg;
 		}
@@ -291,6 +290,12 @@ class Libzot {
 		}
 
 		$m = parse_url($url);
+
+		if (!$m) {
+			logger('zot_refresh: could not parse url');
+			return false;
+		}
+
 		$site_url = unparse_url([ 'scheme' => $m['scheme'], 'host' => $m['host'] ]);
 
 		$s = q("select site_dead from site where site_url = '%s' limit 1",
@@ -323,7 +328,7 @@ class Libzot {
 
 		logger('zot-info: ' . print_r($record, true), LOGGER_DATA, LOG_DEBUG);
 
-		$x = self::import_xchan($record['data'], (($force) ? UPDATE_FLAGS_FORCED : UPDATE_FLAGS_UPDATED));
+		$x = self::import_xchan($record['data']);
 
 		if (!$x['success']) {
 			return false;
@@ -630,23 +635,18 @@ class Libzot {
 	 *   all internal data structures which need to be updated as a result.
 	 *
 	 * @param array $arr => json_decoded discovery packet
-	 * @param int $ud_flags
-	 *    Determines whether to create a directory update record if any changes occur, default is UPDATE_FLAGS_UPDATED
-	 *    $ud_flags = UPDATE_FLAGS_FORCED indicates a forced refresh where we unconditionally create a directory update record
-	 *      this typically occurs once a month for each channel as part of a scheduled ping to notify the directory
-	 *      that the channel still exists
-	 * @param array $ud_arr
-	 *    If set [typically by update_directory_entry()] indicates a specific update table row and more particularly
-	 *    contains a particular address (ud_addr) which needs to be updated in that table.
-	 *
+
 	 * @return array An associative array with:
 	 *   * \e boolean \b success boolean true or false
 	 *   * \e string \b message (optional) error string only if success is false
 	 */
 
-	static function import_xchan($arr, $ud_flags = UPDATE_FLAGS_UPDATED, $ud_arr = null) {
+	static function import_xchan($arr) {
 
-		$ret     = ['success' => false];
+		$ret = [
+			'success' => false,
+			'message' => ''
+		];
 
 		if (!is_array($arr)) {
 			logger('Not an array: ' . print_r($arr, true), LOGGER_DEBUG);
@@ -660,7 +660,7 @@ class Libzot {
 		 */
 		call_hooks('import_xchan', $arr);
 
-		$dirmode = intval(get_config('system', 'directory_mode'));
+		$dirmode = intval(get_config('system', 'directory_mode', DIRECTORY_MODE_NORMAL));
 
 		$changed = false;
 		$what    = '';
@@ -678,15 +678,11 @@ class Libzot {
 		$sig_methods = ((array_key_exists('signing', $arr) && is_array($arr['signing'])) ? $arr['signing'] : ['sha256']);
 		$verified    = false;
 
-		if (!self::verify($arr['id'], $arr['id_sig'], $arr['public_key'])) {
-			logger('Unable to verify channel signature for ' . $arr['primary_location']['address']);
-			return $ret;
-		}
-		else {
+		if (self::verify($arr['id'], $arr['id_sig'], $arr['public_key'])) {
 			$verified = true;
 		}
-
-		if (!$verified) {
+		else {
+			logger('Unable to verify channel signature for ' . $xchan_hash . ' (' . $arr['primary_location']['address']) . ')';
 			$ret['message'] = t('Unable to verify channel signature');
 			return $ret;
 		}
@@ -709,6 +705,7 @@ class Libzot {
 			if ($arr['photo'] && array_key_exists('updated', $arr['photo']) && $arr['photo']['updated'] > $r[0]['xchan_photo_date'])
 				$import_photos = true;
 
+
 			// if we import an entry from a site that's not ours and either or both of us is off the grid - hide the entry.
 			/** @TODO: check if we're the same directory realm, which would mean we are allowed to see it */
 
@@ -725,7 +722,7 @@ class Libzot {
 				$hidden_changed = 1;
 			if (isset($arr['adult_content']) && intval($r[0]['xchan_selfcensored']) != intval($arr['adult_content']))
 				$adult_changed = 1;
-			if (isset($arr['xchan_deleted']) && intval($r[0]['xchan_deleted']) != intval($arr['deleted']))
+			if (isset($arr['deleted']) && intval($r[0]['xchan_deleted']) != intval($arr['deleted']))
 				$deleted_changed = 1;
 
 			// new style 6-MAR-2019
@@ -915,30 +912,26 @@ class Libzot {
 		$s = Libsync::sync_locations($arr, $arr);
 
 		if ($s) {
-			if (isset($s['change_message']))
+			if (!empty($s['change_message']))
 				$what .= $s['change_message'];
-			if (isset($s['changed']))
-				$changed = $s['changed'];
-			if (isset($s['message']))
+			if (!empty($s['changed']))
+				$changed .= $s['changed'];
+			if (!empty($s['message']))
 				$ret['message'] .= $s['message'];
 		}
 
-		// Which entries in the update table are we interested in updating?
-
-		$address = (($ud_arr && $ud_arr['ud_addr']) ? $ud_arr['ud_addr'] : $arr['primary_location']['address']);
-
-
-		// Are we a directory server of some kind?
 
 		$other_realm = false;
 		$realm       = get_directory_realm();
-		if (array_key_exists('site', $arr)
-			&& array_key_exists('realm', $arr['site'])
-			&& (strpos($arr['site']['realm'], $realm) === false))
+		if (array_key_exists('site', $arr) && array_key_exists('realm', $arr['site']) && (strpos($arr['site']['realm'], $realm) === false)) {
 			$other_realm = true;
+		}
 
+		$address = $arr['primary_location']['url'];
 
-		if ($dirmode != DIRECTORY_MODE_NORMAL) {
+		// Are we a directory server of some kind?
+
+		if ($dirmode !== DIRECTORY_MODE_NORMAL) {
 
 			// We're some kind of directory server. However we can only add directory information
 			// if the entry is in the same realm (or is a sub-realm). Sub-realms are denoted by
@@ -946,7 +939,7 @@ class Libzot {
 			// be in directories for the local realm (foo) and also the RED_GLOBAL realm.
 
 			if (array_key_exists('profile', $arr) && is_array($arr['profile']) && (!$other_realm)) {
-				$profile_changed = Libzotdir::import_directory_profile($xchan_hash, $arr['profile'], $address, $ud_flags, 1);
+				$profile_changed = Libzotdir::import_directory_profile($xchan_hash, $arr['profile']);
 				if ($profile_changed) {
 					$what    .= 'profile ';
 					$changed = true;
@@ -972,21 +965,10 @@ class Libzot {
 			}
 		}
 
-		if (($changed) || ($ud_flags == UPDATE_FLAGS_FORCED)) {
-			$guid = random_string() . '@' . \App::get_hostname();
-			Libzotdir::update_modtime($xchan_hash, $guid, $address, $ud_flags);
-			logger('Changed: ' . $what, LOGGER_DEBUG);
-		}
-		elseif (!$ud_flags) {
-			// nothing changed but we still need to update the updates record
-			q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and not (ud_flags & %d) > 0 ",
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($address),
-				intval(UPDATE_FLAGS_UPDATED)
-			);
-		}
+		// update updates if anything changed bump the ud_date
+		Libzotdir::update($xchan_hash, $address, $changed);
 
-		if (!x($ret, 'message')) {
+		if (empty($ret['message'])) {
 			$ret['success'] = true;
 			$ret['hash']    = $xchan_hash;
 		}
@@ -1045,19 +1027,9 @@ class Libzot {
 			}
 
 			if (is_array($x) && array_key_exists('delivery_report', $x) && is_array($x['delivery_report'])) {
-
 				foreach ($x['delivery_report'] as $xx) {
 					call_hooks('dreport_process', $xx);
 					if (is_array($xx) && array_key_exists('message_id', $xx) && DReport::is_storable($xx)) {
-
-						// legacy recipients add a space and their name to the xchan. split those if true.
-						$legacy_recipient = strpos($xx['recipient'], ' ');
-						if ($legacy_recipient !== false) {
-							$legacy_recipient_parts = explode(' ', $xx['recipient'], 2);
-							$xx['recipient']        = $legacy_recipient_parts[0];
-							$xx['name']             = $legacy_recipient_parts[1];
-						}
-
 						q("insert into dreport ( dreport_mid, dreport_site, dreport_recip, dreport_name, dreport_result, dreport_time, dreport_xchan ) values ( '%s', '%s', '%s','%s','%s','%s','%s' ) ",
 							dbesc($xx['message_id']),
 							dbesc($xx['location']),
@@ -1238,7 +1210,6 @@ class Libzot {
 			return;
 		}
 
-
 		if ($has_data) {
 
 			if (in_array($env['type'], ['activity', 'response'])) {
@@ -1248,7 +1219,7 @@ class Libzot {
 					return;
 				}
 
-				$r = q("select hubloc_hash, hubloc_network, hubloc_url from hubloc where hubloc_id_url = '%s' order by hubloc_id desc",
+				$r = q("select hubloc_hash, hubloc_network, hubloc_url from hubloc where hubloc_id_url = '%s' and hubloc_deleted = 0 order by hubloc_id desc",
 					dbesc($AS->actor['id'])
 				);
 
@@ -1256,7 +1227,7 @@ class Libzot {
 					// Author is unknown to this site. Perform channel discovery and try again.
 					$z = discover_by_webbie($AS->actor['id']);
 					if ($z) {
-						$r = q("select hubloc_hash, hubloc_network, hubloc_url from hubloc where hubloc_id_url = '%s' order by hubloc_id desc",
+						$r = q("select hubloc_hash, hubloc_network, hubloc_url from hubloc where hubloc_id_url = '%s' and hubloc_deleted = 0 order by hubloc_id desc",
 							dbesc($AS->actor['id'])
 						);
 					}
@@ -1371,7 +1342,7 @@ class Libzot {
 				return false;
 			}
 			$x = self::find_parent($env, $act);
-			if ($x === $act->id || $x === $act->obj['id']) {
+			if ($x === $act->id || (is_array($act->obj) && array_key_exists('id', $act->obj) && $x === $act->obj['id'])) {
 				return true;
 			}
 		}
@@ -1391,6 +1362,31 @@ class Libzot {
 		return false;
 	}
 
+	static function find_parent_owner_hashes($env, $act) {
+		$r = [];
+		$thread_parent = self::find_parent($env, $act);
+		if ($thread_parent) {
+			$uids = q("SELECT uid FROM item WHERE thr_parent = '%s' OR parent_mid = '%s'",
+				dbesc($thread_parent),
+				dbesc($thread_parent)
+			);
+
+			if ($uids) {
+				$uids = ids_to_querystr($uids, 'uid');
+
+				$z = q("SELECT channel_hash FROM channel WHERE channel_hash != '%s' AND channel_id IN ($uids)",
+					dbesc($env['sender'])
+				);
+
+				if ($z) {
+					foreach ($z as $zv) {
+						$r[] = $zv['channel_hash'];
+					}
+				}
+			}
+		}
+		return $r;
+	}
 
 	/**
 	 * @brief
@@ -1429,7 +1425,9 @@ class Libzot {
 
 		$r = [];
 
-		$c = q("select channel_id, channel_hash from channel where channel_removed = 0");
+		$c = q("select channel_id, channel_hash from channel where channel_hash != '%s' and channel_removed = 0",
+			dbesc($msg['sender'])
+		);
 
 		if ($c) {
 			foreach ($c as $cc) {
@@ -1457,9 +1455,10 @@ class Libzot {
 						if ($tag['type'] === 'Mention' && (strpos($tag['href'], z_root()) !== false)) {
 							$address = basename($tag['href']);
 							if ($address) {
-								$z = q("select channel_hash as hash from channel where channel_address = '%s'
+								$z = q("select channel_hash as hash from channel where channel_address = '%s' and channel_hash != '%s'
 									and channel_removed = 0 limit 1",
-									dbesc($address)
+									dbesc($address),
+									dbesc($msg['sender'])
 								);
 								if ($z) {
 									$r[] = $z[0]['hash'];
@@ -1468,25 +1467,23 @@ class Libzot {
 						}
 					}
 				}
+				if ($act->obj['type'] === 'Tombstone') {
+					// This is a delete. There are no tags to look at - add anyone owning the item.
+					$parent_owners = self::find_parent_owner_hashes($msg, $act);
+					if ($parent_owners) {
+						$r = array_merge($r, $parent_owners);
+					}
+				}
+
 			}
 		}
 		else {
 			// This is a comment. We need to find any parent with ITEM_UPLINK set. But in fact, let's just return
 			// everybody that stored a copy of the parent. This way we know we're covered. We'll check the
 			// comment permissions when we deliver them.
-
-			$thread_parent = self::find_parent($msg, $act);
-
-			if ($thread_parent) {
-				$z = q("select channel_hash as hash from channel left join item on channel.channel_id = item.uid where ( item.thr_parent = '%s' OR item.parent_mid = '%s' ) ",
-					dbesc($thread_parent),
-					dbesc($thread_parent)
-				);
-				if ($z) {
-					foreach ($z as $zv) {
-						$r[] = $zv['hash'];
-					}
-				}
+			$parent_owners = self::find_parent_owner_hashes($msg, $act);
+			if ($parent_owners) {
+				$r = array_merge($r, $parent_owners);
 			}
 		}
 
@@ -1571,9 +1568,14 @@ class Libzot {
 			 * There's a chance the current delivery could take place before the cloned copy arrives
 			 * hence the item could have the wrong ACL and *could* be used in subsequent deliveries or
 			 * access checks.
+			 *
+			 * 30.3.23: block all incoming items from ourselves except if the origin is local.
+			 * This is to prevent multiple relay delivery of items that arrive via sync.
+			 * They have already been relayed at the origin location.
+			 *
 			 */
 
-			if ($sender === $channel['channel_hash'] && $arr['author_xchan'] === $channel['channel_hash'] && $arr['mid'] === $arr['parent_mid']) {
+			if ($sender === $channel['channel_hash'] && $arr['author_xchan'] === $channel['channel_hash'] && !str_starts_with($arr['mid'], z_root())) {
 				$DR->update('self delivery ignored');
 				$result[] = $DR->get();
 				continue;
@@ -1623,13 +1625,23 @@ class Libzot {
 			if ((!$tag_delivery) && (!$local_public)) {
 				$allowed = (perm_is_allowed($channel['channel_id'], $sender, $perm));
 
-				if ((!$allowed) && $perm === 'post_comments') {
-					$parent = q("select * from item where mid = '%s' and uid = %d limit 1",
-						dbesc($arr['parent_mid']),
-						intval($channel['channel_id'])
-					);
-					if ($parent) {
-						$allowed = can_comment_on_post($sender, $parent[0]);
+				$permit_mentions = intval(PConfig::Get($channel['channel_id'], 'system', 'permit_all_mentions') && i_am_mentioned($channel, $arr));
+
+				if (!$allowed) {
+					if ($perm === 'post_comments') {
+						$parent = q("select * from item where mid = '%s' and uid = %d limit 1",
+							dbesc($arr['parent_mid']),
+							intval($channel['channel_id'])
+						);
+						if ($parent) {
+							$allowed = can_comment_on_post($sender, $parent[0]);
+							if (!$allowed && $permit_mentions) {
+								$allowed = true;
+							}
+						}
+
+					} elseif ($permit_mentions) {
+						$allowed = true;
 					}
 				}
 
@@ -1711,7 +1723,7 @@ class Libzot {
 					// this is just an exercise in futility.
 
 					if (perm_is_allowed($channel['channel_id'], $sender, 'send_stream')) {
-						self::fetch_conversation($channel, $arr['parent_mid']);
+						Master::Summon(['Zotconvo', $channel['channel_id'], $arr['parent_mid']]);
 					}
 
 					continue;
@@ -1974,7 +1986,7 @@ class Libzot {
 		$ret = [];
 
 
-		$signer = q("select hubloc_hash, hubloc_url from hubloc where hubloc_id_url = '%s' and hubloc_network = 'zot6' limit 1",
+		$signer = q("select hubloc_hash, hubloc_url from hubloc where hubloc_id_url = '%s' and hubloc_network = 'zot6' order by hubloc_id desc limit 1",
 			dbesc($a['signature']['signer'])
 		);
 
@@ -2002,18 +2014,12 @@ class Libzot {
 				continue;
 			}
 
-			$r = q("select hubloc_hash, hubloc_network from hubloc where hubloc_id_url = '%s'",
-				dbesc($AS->actor['id'])
-			);
-
+			$r = Activity::get_actor_hublocs($AS->actor['id']);
 			$r = self::zot_record_preferred($r);
-
 			if (!$r) {
 				$y = import_author_xchan(['url' => $AS->actor['id']]);
 				if ($y) {
-					$r = q("select hubloc_hash, hubloc_network from hubloc where hubloc_id_url = '%s'",
-						dbesc($AS->actor['id'])
-					);
+					$r = Activity::get_actor_hublocs($AS->actor['id']);
 					$r = self::zot_record_preferred($r);
 				}
 				if (!$r) {
@@ -2023,22 +2029,29 @@ class Libzot {
 			}
 
 			if (isset($AS->obj['actor']['id']) && $AS->obj['actor']['id'] !== $AS->actor['id']) {
-				$y = import_author_xchan(['url' => $AS->obj['actor']['id']]);
-				if (!$y) {
-					logger('FOF Activity: no object actor');
-					continue;
+				$ro = Activity::get_actor_hublocs($AS->obj['actor']['id']);
+				$ro = self::zot_record_preferred($ro);
+				if (!$ro) {
+					$y = import_author_xchan(['url' => $AS->obj['actor']['id']]);
+					if ($y) {
+						$ro = Activity::get_actor_hublocs($AS->obj['actor']['id']);
+						$ro = self::zot_record_preferred($ro);
+					}
+					if (!$ro) {
+						logger('FOF Activity: no obj actor');
+						continue;
+					}
 				}
 			}
 
 			$arr = Activity::decode_note($AS);
 
 			if (!$arr) {
+				logger('FOF Activity: could not decode');
 				continue;
 			}
 
-			if ($r) {
-				$arr['author_xchan'] = $r['hubloc_hash'];
-			}
+			$arr['author_xchan'] = $r['hubloc_hash'];
 
 			if ($signer) {
 				$arr['owner_xchan'] = $signer[0]['hubloc_hash'];
@@ -2306,120 +2319,6 @@ class Libzot {
 	}
 
 
-	/**
-	 * @brief Processes delivery of profile.
-	 *
-	 * @param string $sender
-	 * @param array $arr
-	 * @param array $deliveries (unused)
-	 * @return void
-	 * @see import_directory_profile()
-	 *
-	 */
-	static function process_profile_delivery($sender, $arr, $deliveries) {
-
-		logger('process_profile_delivery', LOGGER_DEBUG);
-
-		$r = q("select xchan_addr from xchan where xchan_hash = '%s' limit 1",
-			dbesc($sender)
-		);
-		if ($r) {
-			Libzotdir::import_directory_profile($sender, $arr, $r[0]['xchan_addr'], UPDATE_FLAGS_UPDATED, 0);
-		}
-	}
-
-
-	/**
-	 * @brief
-	 *
-	 * @param string $sender
-	 * @param array $arr
-	 * @param array $deliveries (unused) deliveries is irrelevant
-	 * @return void
-	 */
-	static function process_location_delivery($sender, $arr, $deliveries) {
-
-		// deliveries is irrelevant
-		logger('process_location_delivery', LOGGER_DEBUG);
-
-		$r = q("select * from xchan where xchan_hash = '%s' limit 1",
-			dbesc($sender)
-		);
-		if ($r) {
-			$xchan = ['id'   => $r[0]['xchan_guid'], 'id_sig' => $r[0]['xchan_guid_sig'],
-					  'hash' => $r[0]['xchan_hash'], 'public_key' => $r[0]['xchan_pubkey']];
-		}
-		if (array_key_exists('locations', $arr) && $arr['locations']) {
-			$x = Libsync::sync_locations($xchan, $arr, true);
-			logger('results: ' . print_r($x, true), LOGGER_DEBUG);
-			if ($x['changed']) {
-				//$guid = random_string() . '@' . App::get_hostname();
-				Libzotdir::update_modtime($sender, $r[0]['xchan_guid'], $arr['locations'][0]['address'], UPDATE_FLAGS_UPDATED);
-			}
-		}
-	}
-
-	/**
-	 * @brief Checks for a moved channel and sets the channel_moved flag.
-	 *
-	 * Currently the effect of this flag is to turn the channel into 'read-only' mode.
-	 * New content will not be processed (there was still an issue with blocking the
-	 * ability to post comments as of 10-Mar-2016).
-	 * We do not physically remove the channel at this time. The hub admin may choose
-	 * to do so, but is encouraged to allow a grace period of several days in case there
-	 * are any issues migrating content. This packet will generally be received by the
-	 * original site when the basic channel import has been processed.
-	 *
-	 * This will only be executed on the old location
-	 * if a new location is reported and there is only one location record.
-	 * The rest of the hubloc syncronisation will be handled within
-	 * sync_locations
-	 *
-	 * @param string $sender_hash A channel hash
-	 * @param array $locations
-	 * @return void
-	 */
-	static function check_location_move($sender_hash, $locations) {
-
-		if (!$locations)
-			return;
-
-		if (count($locations) != 1)
-			return;
-
-		$loc = $locations[0];
-
-		$r = q("select * from channel where channel_hash = '%s' limit 1",
-			dbesc($sender_hash)
-		);
-
-		if (!$r)
-			return;
-
-		if ($loc['url'] !== z_root()) {
-			$x = q("update channel set channel_moved = '%s' where channel_hash = '%s' limit 1",
-				dbesc($loc['url']),
-				dbesc($sender_hash)
-			);
-
-			// federation plugins may wish to notify connections
-			// of the move on singleton networks
-
-			$arr = [
-				'channel'   => $r[0],
-				'locations' => $locations
-			];
-			/**
-			 * @hooks location_move
-			 *   Called when a new location has been provided to a UNO channel (indicating a move rather than a clone).
-			 *   * \e array \b channel
-			 *   * \e array \b locations
-			 */
-			call_hooks('location_move', $arr);
-		}
-	}
-
-
 	/**
 	 * @brief Returns an array with all known distinct hubs for this channel.
 	 *
@@ -2456,19 +2355,6 @@ class Libzot {
 					'deleted'  => (intval($hub['hubloc_deleted']) ? true : false)
 				];
 
-				// version compatibility tweaks
-
-				if (!strpos($z['url_sig'], '.')) {
-					$z['url_sig'] = 'sha256.' . $z['url_sig'];
-				}
-
-				if (!$z['id_url']) {
-					$z['id_url'] = $z['url'] . '/channel/' . substr($z['address'], 0, strpos($z['address'], '@'));
-				}
-				if (!$z['site_id']) {
-					$z['site_id'] = Libzot::make_xchan_hash($z['url'], $z['sitekey']);
-				}
-
 				$ret[] = $z;
 			}
 		}
@@ -2840,12 +2726,12 @@ class Libzot {
 		$e = $r[0];
 		$id = $e['channel_id'];
 
-		$sys_channel     = (intval($e['channel_system']) ? true : false);
+		$sys_channel     = ((empty($e['channel_system'])) ? false : true);
 		$special_channel = (($e['channel_pageflags'] & PAGE_PREMIUM) ? true : false);
 		$adult_channel   = (($e['channel_pageflags'] & PAGE_ADULT) ? true : false);
 		$censored        = (($e['channel_pageflags'] & PAGE_CENSORED) ? true : false);
 		$searchable      = (($e['channel_pageflags'] & PAGE_HIDDEN) ? false : true);
-		$deleted         = (intval($e['xchan_deleted']) ? true : false);
+		$deleted         = ((empty($e['xchan_deleted'])) ? false : true);
 
 		if ($deleted || $censored || $sys_channel)
 			$searchable = false;
@@ -2897,8 +2783,8 @@ class Libzot {
 
 		// Communication details
 
-		$ret['id']     = $e['xchan_guid'];
-		$ret['id_sig'] = self::sign($e['xchan_guid'], $e['channel_prvkey']);
+		$ret['id']     = $e['channel_guid'];
+		$ret['id_sig'] = self::sign($e['channel_guid'], $e['channel_prvkey']);
 
 		$ret['primary_location'] = [
 			'address'         => $e['xchan_addr'],
@@ -2907,10 +2793,10 @@ class Libzot {
 			'follow_url'      => $e['xchan_follow'],
 		];
 
-		$ret['public_key']   = $e['xchan_pubkey'];
+		$ret['public_key']   = $e['channel_pubkey'];
 		$ret['signing_algorithm'] = 'rsa-sha256';
 		$ret['username']     = $e['channel_address'];
-		$ret['name']         = $e['xchan_name'];
+		$ret['name']         = $e['channel_name'];
 		$ret['name_updated'] = $e['xchan_name_date'];
 		$ret['photo']        = [
 			'url'     => $e['xchan_photo_l'],

Zotlabs/Lib/Libzotdir.php

@@ -172,13 +172,12 @@ class Libzotdir {
 	}
 
 	/**
-	 * @brief Checks the directory mode of this hub.
+	 * @brief fetches updates from known directories
 	 *
 	 * Checks the directory mode of this hub to see if it is some form of directory server. If it is,
 	 * get the directory realm of this hub. Fetch a list of all other directory servers in this realm and request
-	 * a directory sync packet. This will contain both directory updates and new ratings. Store these all in the DB.
-	 * In the case of updates, we will query each of them asynchronously from a poller task. Ratings are stored
-	 * directly if the rater's signature matches.
+	 * a directory sync packet. Store these all in the DB.
+	 * In the case of updates, we will query each of them asynchronously from a poller task.
 	 *
 	 * @param int $dirmode;
 	 */
@@ -233,6 +232,8 @@ class Libzotdir {
 		if (! $r)
 			return;
 
+		$dir_trusted_hosts = array_merge(get_directory_fallback_servers(), get_config('system', 'trusted_directory_servers', []));
+
 		foreach ($r as $rr) {
 			if (! $rr['site_directory'])
 				continue;
@@ -264,31 +265,65 @@ class Libzotdir {
 
 			if (is_array($j['transactions']) && count($j['transactions'])) {
 				foreach ($j['transactions'] as $t) {
-
-					if (empty($t['hash']) || empty($t['transaction_id']) || empty($t['address'])) {
+					if (empty($t['hash']) || empty($t['host']) || empty($t['address'])) {
 						continue;
 					}
 
-					$r = q("select * from updates where ud_guid = '%s' limit 1",
-						dbesc($t['transaction_id'])
+					$r = q("select * from updates where ud_hash = '%s' limit 1",
+						dbesc($t['hash'])
 					);
-					if($r)
-						continue;
 
-					$ud_flags = 0;
-					if (is_array($t['flags']) && in_array('deleted',$t['flags']))
-						$ud_flags |= UPDATE_FLAGS_DELETED;
-					if (is_array($t['flags']) && in_array('forced',$t['flags']))
-						$ud_flags |= UPDATE_FLAGS_FORCED;
+					if ($r) {
+						$update = 0;
 
-					$z = q("insert into updates ( ud_hash, ud_guid, ud_date, ud_flags, ud_addr )
-						values ( '%s', '%s', '%s', %d, '%s' ) ",
-						dbesc($t['hash']),
-						dbesc($t['transaction_id']),
-						dbesc($t['timestamp']),
-						intval($ud_flags),
-						dbesc($t['address'])
-					);
+						// no need to look at updates that originated from our own site
+						if ($t['host'] === z_root()) {
+							continue;
+						}
+
+						// there is more recent xchan information
+						if ($r[0]['ud_date'] <= $t['timestamp']) {
+							$update = 1;
+						}
+
+						// the host is trusted and flags have changed - update flags immediately
+						if (in_array($t['host'], $dir_trusted_hosts) &&
+							$rr['site_url'] === $t['host'] &&
+							intval($r[0]['ud_flags']) !== intval($t['flags'])) {
+
+							q("UPDATE updates SET ud_update = %d, ud_flags = %d WHERE ud_id = %d",
+								intval($update),
+								intval($t['flags']),
+								dbesc($r[0]['ud_id'])
+							);
+
+							q("UPDATE xchan SET xchan_censored = %d WHERE xchan_hash = '%s'",
+								intval($t['flags']),
+								dbesc($r[0]['ud_hash'])
+							);
+
+							continue;
+						}
+
+						if (!$update) {
+							continue;
+						}
+
+						q("UPDATE updates SET ud_update = %d WHERE ud_id = %d",
+							intval($update),
+							dbesc($r[0]['ud_id'])
+						);
+					}
+					else {
+						q("insert into updates ( ud_hash, ud_host, ud_date, ud_addr, ud_update, ud_flags )
+							values ( '%s', '%s', '%s', '%s', 1, %d) ",
+							dbesc($t['hash']),
+							dbesc($t['host']),
+							dbesc($t['timestamp']),
+							dbesc($t['address']),
+							dbesc(in_array($t['host'], $dir_trusted_hosts) ? $t['flags'] : 0)
+						);
+					}
 				}
 			}
 		}
@@ -303,8 +338,9 @@ class Libzotdir {
 	 *
 	 * Ignore updating records marked as deleted.
 	 *
-	 * If successful, sets ud_last in the DB to the current datetime for this
+	 * If successful, sets ud_updated in the DB to the current datetime for this
 	 * reddress/webbie.
+	 * Else update ud_last so we can stop trying after 7 days (Daemon/Poller.php)
 	 *
 	 * @param array $ud Entry from update table
 	 */
@@ -313,32 +349,47 @@ class Libzotdir {
 
 		logger('update_directory_entry: ' . print_r($ud,true), LOGGER_DATA);
 
-		if ($ud['ud_addr'] && (! ($ud['ud_flags'] & UPDATE_FLAGS_DELETED))) {
-			$success = false;
-			$zf = [];
+		// TODO: remove this check after all directory servers have version > 8.4
+		// ud_addr will always be the channel url at that time
+		$href = ((strpos($ud['ud_addr'], '://') === false) ? Webfinger::zot_url(punify($ud['ud_addr'])) : punify($ud['ud_addr']));
+		if($href) {
+			$zf = Zotfinger::exec($href);
+			if($zf && array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $href && intval($zf['signature']['header_valid'])) {
+				$xc = Libzot::import_xchan($zf['data']);
+
+				// xchan_hash mismatch - this can happen after a site re-install at the same url
+				if ($xc['success'] && $xc['hash'] !== $ud['ud_hash']) {
+					self::delete_by_hash($ud['ud_hash']);
+				}
+
+				// backwards compatibility: Libzot::import_xchan(), where self::update() is called,
+				// will fail with versions < 8.4 if the channel has been locally deleted.
+				// In this case we will update the updates record here without bumping the date
+				// since we could not verify if anything changed.
+				if (!$xc['success'] && !empty($zf['data']['deleted_locally'])) {
+					self::update($ud['ud_hash'], $ud['ud_addr'], false);
+				}
 
-			$href = Webfinger::zot_url(punify($ud['ud_addr']));
-			if($href) {
-				$zf = Zotfinger::exec($href);
-			}
-			if(array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $href && intval($zf['signature']['header_valid'])) {
-				$xc = Libzot::import_xchan($zf['data'], 0, $ud);
 				// This is a workaround for a missing xchan_updated column
 				// TODO: implement xchan_updated in the xchan table and update this column instead
-				if($zf['data']['primary_location']['address'] && $zf['data']['primary_location']['url']) {
+				if(!empty($zf['data']['primary_location']['url'])) {
 					q("UPDATE hubloc SET hubloc_updated = '%s' WHERE hubloc_id_url = '%s' AND hubloc_primary = 1",
 						dbesc(datetime_convert()),
 						dbesc($zf['data']['primary_location']['url'])
 					);
 				}
-			}
-			else {
-				q("update updates set ud_last = '%s' where ud_addr = '%s'",
-					dbesc(datetime_convert()),
-					dbesc($ud['ud_addr'])
-				);
+
+				return true;
 			}
 		}
+
+		q("UPDATE updates SET ud_addr = '%s', ud_last = '%s' WHERE ud_hash = '%s'",
+			dbesc($href ? $href : $ud['ud_addr']),
+			dbesc(datetime_convert()),
+			dbesc($ud['ud_hash'])
+		);
+
+		return false;
 	}
 
 
@@ -353,85 +404,78 @@ class Libzotdir {
 	 */
 
 	static function local_dir_update($uid, $force) {
+		logger('local_dir_update uid: ' . $uid, LOGGER_DEBUG);
 
-
-		logger('local_dir_update: uid: ' . $uid, LOGGER_DEBUG);
-
-		$p = q("select channel.channel_hash, channel_address, channel_timezone, profile.* from profile left join channel on channel_id = uid where uid = %d and is_default = 1",
+		$p = q("select channel.channel_hash, channel_address, channel_timezone, profile.*, xchan.xchan_hidden, xchan.xchan_url from profile left join channel on channel_id = uid left join xchan on channel_hash = xchan_hash where profile.uid = %d and profile.is_default = 1",
 			intval($uid)
 		);
 
-		$profile = array();
-		$profile['encoding'] = 'zot';
-
-		if ($p) {
-			$hash = $p[0]['channel_hash'];
-
-			$profile['description'] = $p[0]['pdesc'];
-			$profile['birthday']    = $p[0]['dob'];
-			if ($age = age($p[0]['dob'],$p[0]['channel_timezone'],''))
-				$profile['age'] = $age;
-
-			$profile['gender']      = $p[0]['gender'];
-			$profile['marital']     = $p[0]['marital'];
-			$profile['sexual']      = $p[0]['sexual'];
-			$profile['locale']      = $p[0]['locality'];
-			$profile['region']      = $p[0]['region'];
-			$profile['postcode']    = $p[0]['postal_code'];
-			$profile['country']     = $p[0]['country_name'];
-			$profile['about']       = $p[0]['about'];
-			$profile['homepage']    = $p[0]['homepage'];
-			$profile['hometown']    = $p[0]['hometown'];
-
-			if ($p[0]['keywords']) {
-				$tags = array();
-				$k = explode(' ', $p[0]['keywords']);
-				if ($k)
-					foreach ($k as $kk)
-						if (trim($kk))
-							$tags[] = trim($kk);
-
-				if ($tags)
-					$profile['keywords'] = $tags;
-			}
-
-			$hidden = (1 - intval($p[0]['publish']));
-
-			logger('hidden: ' . $hidden);
-
-			$r = q("select xchan_hidden from xchan where xchan_hash = '%s'",
-				dbesc($p[0]['channel_hash'])
-			);
-
-			if(intval($r[0]['xchan_hidden']) != $hidden) {
-				$r = q("update xchan set xchan_hidden = %d where xchan_hash = '%s'",
-					intval($hidden),
-					dbesc($hash)
-				);
-			}
-
-			$arr = [ 'channel_id' => $uid, 'hash' => $hash, 'profile' => $profile ];
-			call_hooks('local_dir_update', $arr);
-
-			$address = channel_reddress($p[0]);
-
-			if (perm_is_allowed($uid, '', 'view_profile')) {
-				self::import_directory_profile($hash, $arr['profile'], $address, 0);
-			}
-			else {
-				// they may have made it private
-				q("delete from xprof where xprof_hash = '%s'",
-					dbesc($hash)
-				);
-				q("delete from xtag where xtag_hash = '%s'",
-					dbesc($hash)
-				);
-			}
-
+		if (!$p) {
+			logger('profile not found');
+			return;
 		}
 
-		$ud_hash = random_string() . '@' . \App::get_hostname();
-		self::update_modtime($hash, $ud_hash, channel_reddress($p[0]),(($force) ? UPDATE_FLAGS_FORCED : UPDATE_FLAGS_UPDATED));
+		$profile = [];
+		$profile['encoding'] = 'zot';
+
+		$hash = $p[0]['channel_hash'];
+
+		$profile['description'] = $p[0]['pdesc'];
+		$profile['birthday']    = $p[0]['dob'];
+		if ($age = age($p[0]['dob'],$p[0]['channel_timezone'],''))
+			$profile['age'] = $age;
+
+		$profile['gender']      = $p[0]['gender'];
+		$profile['marital']     = $p[0]['marital'];
+		$profile['sexual']      = $p[0]['sexual'];
+		$profile['locale']      = $p[0]['locality'];
+		$profile['region']      = $p[0]['region'];
+		$profile['postcode']    = $p[0]['postal_code'];
+		$profile['country']     = $p[0]['country_name'];
+		$profile['about']       = $p[0]['about'];
+		$profile['homepage']    = $p[0]['homepage'];
+		$profile['hometown']    = $p[0]['hometown'];
+
+		if ($p[0]['keywords']) {
+			$tags = array();
+			$k = explode(' ', $p[0]['keywords']);
+			if ($k)
+				foreach ($k as $kk)
+					if (trim($kk))
+						$tags[] = trim($kk);
+
+			if ($tags)
+				$profile['keywords'] = $tags;
+		}
+
+		$hidden = (1 - intval($p[0]['publish']));
+
+		logger('hidden: ' . $hidden);
+
+		if(intval($p[0]['xchan_hidden']) !== $hidden) {
+			q("update xchan set xchan_hidden = %d where xchan_hash = '%s'",
+				intval($hidden),
+				dbesc($hash)
+			);
+		}
+
+		$arr = [ 'channel_id' => $uid, 'hash' => $hash, 'profile' => $profile ];
+		call_hooks('local_dir_update', $arr);
+
+		if (perm_is_allowed($uid, '', 'view_profile')) {
+			self::import_directory_profile($hash, $arr['profile']);
+		}
+		else {
+			// they may have made it private
+			q("delete from xprof where xprof_hash = '%s'",
+				dbesc($hash)
+			);
+			q("delete from xtag where xtag_hash = '%s'",
+				dbesc($hash)
+			);
+		}
+
+		self::update($hash, $p[0]['xchan_url']);
 	}
 
 
@@ -441,13 +485,10 @@ class Libzotdir {
 	 *
 	 * @param string $hash
 	 * @param array $profile
-	 * @param string $addr
-	 * @param number $ud_flags (optional) UPDATE_FLAGS_UPDATED
-	 * @param number $suppress_update (optional) default 0
 	 * @return boolean $updated if something changed
 	 */
 
-	static function import_directory_profile($hash, $profile, $addr, $ud_flags = UPDATE_FLAGS_UPDATED, $suppress_update = 0) {
+	static function import_directory_profile($hash, $profile) {
 
 		logger('import_directory_profile', LOGGER_DEBUG);
 		if (! $hash)
@@ -480,7 +521,12 @@ class Libzotdir {
 		$clean = array();
 		if (array_key_exists('keywords', $profile) and is_array($profile['keywords'])) {
 			self::import_directory_keywords($hash,$profile['keywords']);
+
 			foreach ($profile['keywords'] as $kw) {
+				if (in_array($kw, $clean)) {
+					continue;
+				}
+
 				$kw = trim(htmlspecialchars($kw,ENT_COMPAT, 'UTF-8', false));
 				$kw = trim($kw, ',');
 				$clean[] = $kw;
@@ -587,9 +633,6 @@ class Libzotdir {
 		 */
 		call_hooks('import_directory_profile', $d);
 
-		if (($d['update']) && (! $suppress_update))
-			self::update_modtime($arr['xprof_hash'],random_string() . '@' . \App::get_hostname(), $addr, $ud_flags);
-
 		return $d['update'];
 	}
 
@@ -607,6 +650,10 @@ class Libzotdir {
 			dbesc($hash)
 		);
 
+		$xchan = q("select xchan_censored from xchan where xchan_hash = '%s'",
+			dbesc($hash)
+		);
+
 		if($r) {
 			foreach($r as $rr)
 				$existing[] = $rr['xtag_term'];
@@ -614,6 +661,10 @@ class Libzotdir {
 
 		$clean = array();
 		foreach($keywords as $kw) {
+			if (in_array($kw, $clean)) {
+				continue;
+			}
+
 			$kw = trim(htmlspecialchars($kw,ENT_COMPAT, 'UTF-8', false));
 			$kw = trim($kw, ',');
 			$clean[] = $kw;
@@ -628,9 +679,10 @@ class Libzotdir {
 		}
 		foreach($clean as $x) {
 			if(! in_array($x, $existing)) {
-				$r = q("insert into xtag ( xtag_hash, xtag_term, xtag_flags) values ( '%s' ,'%s', 0 )",
+				$r = q("insert into xtag ( xtag_hash, xtag_term, xtag_flags) values ( '%s' ,'%s', %d )",
 					dbesc($hash),
-					dbesc($x)
+					dbesc($x),
+					intval($xchan[0]['xchan_censored'])
 				);
 			}
 		}
@@ -640,13 +692,12 @@ class Libzotdir {
 	/**
 	 * @brief
 	 *
-	 * @param string $hash
-	 * @param string $guid
-	 * @param string $addr
-	 * @param int $flags (optional) default 0
+	 * @param string $hash the channel hash
+	 * @param string $addr the channel url
+	 * @param bool $bump_date (optional) default true
 	 */
 
-	static function update_modtime($hash, $guid, $addr, $flags = 0) {
+	static function update($hash, $addr, $bump_date = true, $flag = null) {
 
 		$dirmode = intval(get_config('system', 'directory_mode'));
 
@@ -654,26 +705,70 @@ class Libzotdir {
 			return;
 		}
 
-		if (empty($hash) || empty($guid) || empty($addr)) {
+		if (empty($hash) || empty($addr)) {
 			return;
 		}
 
-		if($flags) {
-			q("insert into updates (ud_hash, ud_guid, ud_date, ud_flags, ud_addr ) values ( '%s', '%s', '%s', %d, '%s' )",
-				dbesc($hash),
-				dbesc($guid),
-				dbesc(datetime_convert()),
-				intval($flags),
-				dbesc($addr)
-			);
+		$u = q("SELECT * FROM updates WHERE ud_hash = '%s' LIMIT 1",
+			dbesc($hash)
+		);
+
+		$date_sql = '';
+		if ($bump_date) {
+			$date_sql = "ud_date = '" . dbesc(datetime_convert()) . "',";
 		}
-		else {
-			q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and not (ud_flags & %d)>0 ",
-				intval(UPDATE_FLAGS_UPDATED),
+
+		$flag_sql = '';
+		if ($flag !== null) {
+			$flag_sql = "ud_flags = '" . intval($flag) . "',";
+		}
+
+
+		if ($u) {
+			$x = q("UPDATE updates SET $date_sql $flag_sql ud_last = '%s', ud_host = '%s', ud_addr = '%s', ud_update = 0 WHERE ud_id = %d",
+				dbesc(NULL_DATE),
+				dbesc(z_root()),
 				dbesc($addr),
-				intval(UPDATE_FLAGS_UPDATED)
+				intval($u[0]['ud_id'])
 			);
+
+			return;
 		}
+
+		q("INSERT INTO updates (ud_hash, ud_host, ud_date, ud_addr, ud_flags) VALUES ( '%s', '%s', '%s', '%s', %d )",
+			dbesc($hash),
+			dbesc(z_root()),
+			dbesc(datetime_convert()),
+			dbesc($addr),
+			intval($flag)
+		);
+
+		return;
+
+	}
+
+
+	/**
+	 * @brief deletes a entry in updates by hash
+	 *
+	 * @param string $hash the channel hash
+	 * @return boolean
+	 */
+
+	static function delete_by_hash($hash) {
+		if (!$hash) {
+			return false;
+		}
+
+		$x = q("DELETE FROM updates WHERE ud_hash = '%s'",
+			dbesc($hash)
+		);
+
+		if ($x) {
+			return true;
+		}
+
+		return false;
 	}
 
 }

Zotlabs/Lib/MessageFilter.php

@@ -25,7 +25,7 @@ class MessageFilter {
 
 		if ($exclude) {
 			foreach ($exclude as $word) {
-				$word = trim($word);
+				$word = html_entity_decode(trim($word));
 				if (! $word) {
 					continue;
 				}
@@ -73,7 +73,7 @@ class MessageFilter {
 
 		if ($include) {
 			foreach ($include as $word) {
-				$word = trim($word);
+				$word = html_entity_decode(trim($word));
 				if (! $word) {
 					continue;
 				}

Zotlabs/Lib/Queue.php

@@ -57,7 +57,6 @@ class Queue {
 			outq_priority = outq_priority + %d,
 			outq_scheduled = '%s'
 			WHERE outq_hash = '%s'",
-
 			dbesc(datetime_convert()),
 			intval($add_priority),
 			dbesc($next),
@@ -65,16 +64,38 @@ class Queue {
 		);
 	}
 
-
-	static function remove($id,$channel_id = 0) {
-		logger('queue: remove queue item ' . $id,LOGGER_DEBUG);
+	public static function remove($id, $channel_id = 0) {
+		logger('queue: remove queue item ' . $id, LOGGER_DEBUG);
 		$sql_extra = (($channel_id) ? " and outq_channel = " . intval($channel_id) . " " : '');
 
-		q("DELETE FROM outq WHERE outq_hash = '%s' $sql_extra",
+		// figure out what endpoint it is going to.
+		$record = q("select outq_posturl from outq where outq_hash = '%s' $sql_extra",
 			dbesc($id)
 		);
-	}
 
+		if ($record) {
+			q("DELETE FROM outq WHERE outq_hash = '%s' $sql_extra",
+				dbesc($id)
+			);
+
+			// If there's anything remaining in the queue for this site, move one of them to the next active
+			// queue run by setting outq_scheduled back to the present. We may be attempting to deliver it
+			// as a 'piled_up' delivery, but this ensures the site has an active queue entry as long as queued
+			// entries still exist for it. This fixes an issue where one immediate delivery left everything
+			// else for that site undeliverable since all the other entries had been pushed far into the future.
+
+			$r = q("SELECT outq_hash, outq_posturl FROM outq WHERE outq_posturl = '%s' LIMIT 1",
+				dbesc($record[0]['outq_posturl'])
+			);
+
+			if ($r) {
+				$hashes = ids_to_querystr($r, 'outq_hash', true);
+				$x = q("UPDATE outq SET outq_scheduled = '%s' WHERE outq_hash IN ($hashes)",
+					dbesc(datetime_convert())
+				);
+			}
+		}
+	}
 
 	static function remove_by_posturl($posturl) {
 		logger('queue: remove queue posturl ' . $posturl,LOGGER_DEBUG);
@@ -84,8 +105,6 @@ class Queue {
 		);
 	}
 
-
-
 	static function set_delivered($id,$channel = 0) {
 		logger('queue: set delivered ' . $id,LOGGER_DEBUG);
 		$sql_extra = (($channel['channel_id']) ? " and outq_channel = " . intval($channel['channel_id']) . " " : '');
@@ -152,17 +171,19 @@ class Queue {
 			$y = q("select site_update, site_dead from site where site_url = '%s' ",
 				dbesc($base)
 			);
-			if($y) {
-				if(intval($y[0]['site_dead'])) {
+
+			if ($y) {
+				// Don't bother delivering if the site is dead.
+				// And if we haven't heard from the site in over a month - let them through but 3 strikes you're out.
+				if (intval($y[0]['site_dead']) || ($y[0]['site_update'] < datetime_convert('UTC', 'UTC', 'now - 1 month') && $outq['outq_priority'] > 20)) {
+					q("update dreport set dreport_result = '%s' where dreport_queue = '%s'",
+						dbesc('site dead'),
+						dbesc($outq['outq_hash'])
+					);
 					self::remove_by_posturl($outq['outq_posturl']);
 					logger('dead site ignored ' . $base);
 					return;
 				}
-				if($y[0]['site_update'] < datetime_convert('UTC','UTC','now - 1 month')) {
-					self::update($outq['outq_hash'], 10);
-					logger('immediate delivery deferred for site ' . $base);
-					return;
-				}
 			}
 			else {
 
@@ -225,7 +246,7 @@ class Queue {
 
 			if($result['success']) {
 				logger('deliver: remote zot delivery succeeded to ' . $outq['outq_posturl']);
-				Libzot::process_response($outq['outq_posturl'],$result, $outq);
+				Libzot::process_response($outq['outq_posturl'], $result, $outq);
 			}
 			else {
 				logger('deliver: remote zot delivery failed to ' . $outq['outq_posturl']);

Zotlabs/Lib/QueueWorker.php

@@ -0,0 +1,378 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+use Ramsey\Uuid\Uuid;
+use Ramsey\Uuid\Exception\UnableToBuildUuidException;
+
+class QueueWorker {
+
+	public static $queueworker = null;
+	public static $maxworkers = 0;
+	public static $workermaxage = 0;
+	public static $workersleep = 100;
+	public static $default_priorities = [
+		'Notifier'         => 10,
+		'Deliver'          => 10,
+		'Cache_query'      => 10,
+		'Content_importer' => 1,
+		'File_importer'    => 1,
+		'Channel_purge'    => 1,
+		'Directory'        => 1
+	];
+
+	// Exceptions for processtimeout ($workermaxage) value.
+	// Currently the value is overriden with 3600 seconds (1h).
+	public static $long_running_cmd = [
+		'Queue',
+		'Expire'
+	];
+
+	private static function qstart() {
+		q('START TRANSACTION');
+	}
+
+	private static function qcommit() {
+		q("COMMIT");
+	}
+
+	private static function qrollback() {
+		q("ROLLBACK");
+	}
+
+	public static function Summon($argv) {
+
+		if ($argv[0] !== 'Queueworker') {
+
+			$priority = 0; // @TODO allow reprioritization
+
+			if (isset(self::$default_priorities[$argv[0]])) {
+				$priority = self::$default_priorities[$argv[0]];
+			}
+
+			$workinfo      = ['argc' => count($argv), 'argv' => $argv];
+			$workinfo_json = json_encode($workinfo);
+			$uuid          = self::getUuid($workinfo_json);
+
+			$r = q("SELECT * FROM workerq WHERE workerq_uuid = '%s'",
+				dbesc($uuid)
+			);
+			if ($r) {
+				logger("Summon: Ignoring duplicate workerq task", LOGGER_DEBUG);
+				logger(print_r($workinfo, true));
+				return;
+			}
+
+			self::qstart();
+			$r = q("INSERT INTO workerq (workerq_priority, workerq_data, workerq_uuid, workerq_cmd) VALUES (%d, '%s', '%s', '%s')",
+				intval($priority),
+				$workinfo_json,
+				dbesc($uuid),
+				dbesc($argv[0])
+			);
+			if (!$r) {
+				self::qrollback();
+				logger("INSERT FAILED", LOGGER_DEBUG);
+				return;
+			}
+			self::qcommit();
+			logger('INSERTED: ' . $workinfo_json, LOGGER_DEBUG);
+		}
+
+		$workers = self::GetWorkerCount();
+		if ($workers < self::$maxworkers) {
+			logger($workers . '/' . self::$maxworkers . ' workers active', LOGGER_DEBUG);
+			$phpbin = get_config('system', 'phpbin', 'php');
+			proc_run($phpbin, 'Zotlabs/Daemon/Master.php', ['Queueworker']);
+		}
+	}
+
+	public static function Release($argv) {
+
+		if ($argv[0] !== 'Queueworker') {
+
+			$priority = 0; // @TODO allow reprioritization
+			if (isset(self::$default_priorities[$argv[0]])) {
+				$priority = self::$default_priorities[$argv[0]];
+			}
+
+			$workinfo      = ['argc' => count($argv), 'argv' => $argv];
+			$workinfo_json = json_encode($workinfo);
+			$uuid          = self::getUuid($workinfo_json);
+
+			$r = q("SELECT * FROM workerq WHERE workerq_uuid = '%s'",
+				dbesc($uuid)
+			);
+			if ($r) {
+				logger("Release: Duplicate task - do not insert.", LOGGER_DEBUG);
+				logger(print_r($workinfo, true));
+				return;
+			}
+
+			self::qstart();
+			$r = q("INSERT INTO workerq (workerq_priority, workerq_data, workerq_uuid, workerq_cmd) VALUES (%d, '%s', '%s', '%s')",
+				intval($priority),
+				$workinfo_json,
+				dbesc($uuid),
+				dbesc($argv[0])
+			);
+			if (!$r) {
+				self::qrollback();
+				logger("Insert failed: " . $workinfo_json, LOGGER_DEBUG);
+				return;
+			}
+			self::qcommit();
+			logger('INSERTED: ' . $workinfo_json, LOGGER_DEBUG);
+		}
+
+		self::Process();
+	}
+
+	public static function GetWorkerCount() {
+		if (self::$maxworkers == 0) {
+			self::$maxworkers = get_config('queueworker', 'max_queueworkers', 4);
+			self::$maxworkers = self::$maxworkers > 3 ? self::$maxworkers : 4;
+		}
+		if (self::$workermaxage == 0) {
+			self::$workermaxage = get_config('queueworker', 'max_queueworker_age');
+			self::$workermaxage = self::$workermaxage > 120 ? self::$workermaxage : 300;
+		}
+
+		self::qstart();
+
+		// skip locked is preferred but is not supported by mariadb < 10.6 which is still used a lot - hence make it optional
+		$sql_quirks = ((get_config('system', 'db_skip_locked_supported')) ? 'SKIP LOCKED' : 'NOWAIT');
+
+		$r = q("SELECT workerq_id FROM workerq WHERE workerq_reservationid IS NOT NULL AND workerq_processtimeout < %s FOR UPDATE $sql_quirks",
+			db_utcnow()
+		);
+
+		if ($r) {
+			// TODO: some long running services store their pid in config.procid.daemon
+			// we could possibly check if a pid exist and check if the process is still alive
+			// prior to reseting workerq_reservationid
+
+			$ids = ids_to_querystr($r, 'workerq_id');
+			$u = dbq("update workerq set workerq_reservationid = null where workerq_id in ($ids)");
+		}
+
+		self::qcommit();
+
+		//q("update workerq set workerq_reservationid = null where workerq_reservationid is not null and workerq_processtimeout < %s",
+			//db_utcnow()
+		//);
+
+		//usleep(self::$workersleep);
+
+		$workers = dbq("select count(*) as total from workerq where workerq_reservationid is not null");
+		logger("WORKERCOUNT: " . $workers[0]['total'], LOGGER_DEBUG);
+
+		return intval($workers[0]['total']);
+	}
+
+	public static function GetWorkerID() {
+		if (self::$queueworker) {
+			return self::$queueworker;
+		}
+
+		$wid = uniqid('', true);
+
+		//usleep(mt_rand(300000, 1000000)); //Sleep .3 - 1 seconds before creating a new worker.
+
+		$workers = self::GetWorkerCount();
+
+		if ($workers >= self::$maxworkers) {
+			logger("Too many active workers ($workers) max = " . self::$maxworkers, LOGGER_DEBUG);
+			return false;
+		}
+
+		self::$queueworker = $wid;
+
+		return $wid;
+	}
+
+	private static function getWorkId() {
+		self::GetWorkerCount();
+
+		self::qstart();
+
+		// skip locked is preferred but is not supported by mariadb < 10.6 which is still used a lot - hence make it optional
+		$sql_quirks = ((get_config('system', 'db_skip_locked_supported')) ? 'SKIP LOCKED' : 'NOWAIT');
+
+		$work = dbq("SELECT workerq_id, workerq_cmd FROM workerq WHERE workerq_reservationid IS NULL ORDER BY workerq_priority DESC, workerq_id ASC LIMIT 1 FOR UPDATE $sql_quirks");
+
+		if (!$work) {
+			self::qrollback();
+			return false;
+		}
+
+		$id = $work[0]['workerq_id'];
+		$cmd = $work[0]['workerq_cmd'];
+		$age = self::$workermaxage;
+
+		if (in_array($cmd, self::$long_running_cmd)) {
+			$age = 3600; // 1h TODO: make this configurable
+		}
+
+		$work = q("UPDATE workerq SET workerq_reservationid = '%s', workerq_processtimeout = %s + INTERVAL %s WHERE workerq_id = %d",
+			self::$queueworker,
+			db_utcnow(),
+			db_quoteinterval($age . " SECOND"),
+			intval($id)
+		);
+
+		if (!$work) {
+			self::qrollback();
+			logger("Could not update workerq.", LOGGER_DEBUG);
+			return false;
+		}
+
+		logger("GOTWORK: " . json_encode($work), LOGGER_DEBUG);
+		self::qcommit();
+
+		return $id;
+	}
+
+	public static function Process() {
+		$sleep = intval(get_config('queueworker', 'queue_worker_sleep', 100));
+		$auto_queue_worker_sleep = get_config('queueworker', 'auto_queue_worker_sleep', 0);
+
+		if (!self::GetWorkerID()) {
+			if ($auto_queue_worker_sleep) {
+				set_config('queueworker', 'queue_worker_sleep', $sleep + 100);
+			}
+
+			logger('Unable to get worker ID. Exiting.', LOGGER_DEBUG);
+			killme();
+		}
+
+		if ($auto_queue_worker_sleep && $sleep > 100) {
+			$next_sleep = $sleep - 100;
+			set_config('queueworker', 'queue_worker_sleep', (($next_sleep < 100) ? 100 : $next_sleep));
+		}
+
+		$jobs               = 0;
+		$workid             = self::getWorkId();
+		$load_average_sleep = false;
+		self::$workersleep  = $sleep;
+		self::$workersleep  = ((intval(self::$workersleep) > 100) ? intval(self::$workersleep) : 100);
+
+		if (function_exists('sys_getloadavg') && get_config('queueworker', 'load_average_sleep')) {
+			// very experimental!
+			$load_average_sleep = true;
+		}
+
+		while ($workid) {
+
+			if ($load_average_sleep) {
+				$load_average      = sys_getloadavg();
+				self::$workersleep = intval($load_average[0]) * 10000;
+
+				if (!self::$workersleep) {
+					self::$workersleep = 100;
+				}
+			}
+
+			logger('queue_worker_sleep: ' . self::$workersleep, LOGGER_DEBUG);
+
+			usleep(self::$workersleep);
+
+			$workitem = dbq("SELECT * FROM workerq WHERE workerq_id = $workid");
+
+			if ($workitem) {
+				// At least SOME work to do.... in case there's more, let's ramp up workers.
+				$workers = self::GetWorkerCount();
+
+				if ($workers < self::$maxworkers) {
+					logger($workers . '/' . self::$maxworkers . ' workers active', LOGGER_DEBUG);
+					$phpbin = get_config('system', 'phpbin', 'php');
+					proc_run($phpbin, 'Zotlabs/Daemon/Master.php', ['Queueworker']);
+				}
+
+				$jobs++;
+
+				logger("Workinfo: " . $workitem[0]['workerq_data'], LOGGER_DEBUG);
+
+				$workinfo = json_decode($workitem[0]['workerq_data'], true);
+				$argv     = $workinfo['argv'];
+
+				$cls  = '\\Zotlabs\\Daemon\\' . $argv[0];
+				$argv = flatten_array_recursive($argv);
+				$argc = count($argv);
+				$rnd = random_string();
+
+				logger('PROCESSING: ' . $rnd . ' ' . print_r($argv[0], true));
+
+				$cls::run($argc, $argv);
+
+				logger('COMPLETED: ' . $rnd);
+
+				// @FIXME: Right now we assume that if we get a return, everything is OK.
+				// At some point we may want to test whether the run returns true/false
+				// and requeue the work to be tried again if needed.  But we probably want
+				// to implement some sort of "retry interval" first.
+
+				dbq("delete from workerq where workerq_id = $workid");
+			}
+			else {
+				logger("NO WORKITEM!", LOGGER_DEBUG);
+			}
+			$workid = self::getWorkId();
+		}
+		logger('Master: Worker Thread: queue items processed:' . $jobs, LOGGER_DEBUG);
+	}
+
+	public static function ClearQueue() {
+		$work = q("select * from workerq");
+		while ($work) {
+			foreach ($work as $workitem) {
+				$workinfo = json_decode($workitem['v'], true);
+				$argc     = $workinfo['argc'];
+				$argv     = $workinfo['argv'];
+
+				logger('Master: process: ' . print_r($argv, true), LOGGER_ALL, LOG_DEBUG);
+
+				if (!isset($argv[0])) {
+					q("delete from workerq where workerq_id = %d",
+						$work[0]['workerq_id']
+					);
+					continue;
+				}
+
+				$cls = '\\Zotlabs\\Daemon\\' . $argv[0];
+				$cls::run($argc, $argv);
+
+				q("delete from workerq where workerq_id = %d",
+					$work[0]['workerq_id']
+				);
+
+				//Give the server .3 seconds to catch its breath between tasks.
+				//This will hopefully keep it from crashing to it's knees entirely
+				//if the last task ended up initiating other parallel processes
+				//(eg. polling remotes)
+				usleep(300000);
+			}
+
+			//Make sure nothing new came in
+			$work = q("select * from workerq");
+		}
+	}
+
+	/**
+	 * @brief Generate a name-based v5 UUID with custom namespace
+	 *
+	 * @param string $data
+	 * @return string $uuid
+	 */
+	private static function getUuid(string $data) {
+		$namespace = '3a112e42-f147-4ccf-a78b-f6841339ea2a';
+		try {
+			$uuid = Uuid::uuid5($namespace, $data)->toString();
+		} catch (UnableToBuildUuidException $e) {
+			logger('UUID generation failed');
+			return '';
+		}
+		return $uuid;
+	}
+
+}

Zotlabs/Lib/ThreadItem.php

@@ -137,13 +137,7 @@ class ThreadItem {
 			$shareable = false;
 		}
 
-		$privacy_warning = false;
-		if(intval($item['item_private']) && ($item['owner']['xchan_network'] === 'activitypub')) {
-			$recips = get_iconfig($item['parent'], 'activitypub', 'recips');
-
-			if(! is_array($recips['to']) || ! in_array($observer['xchan_url'], $recips['to']))
-				$privacy_warning = true;
-		}
+		$privacy_warning = ($item['owner']['xchan_network'] === 'activitypub' && intval($item['item_private']) === 1);
 
 		if ($lock) {
  			if (($item['mid'] == $item['parent_mid']) && isset($item['term']) && count(get_terms_oftype($item['term'], TERM_FORUM))) {
@@ -203,7 +197,7 @@ class ThreadItem {
 		$response_verbs = array('like');
 		if(feature_enabled($conv->get_profile_owner(),'dislike'))
 			$response_verbs[] = 'dislike';
-		if($item['obj_type'] === ACTIVITY_OBJ_EVENT) {
+		if(in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT])) {
 			$response_verbs[] = 'attendyes';
 			$response_verbs[] = 'attendno';
 			$response_verbs[] = 'attendmaybe';
@@ -305,7 +299,7 @@ class ThreadItem {
 		}
 
 		$has_event = false;
-		if(($item['obj_type'] === ACTIVITY_OBJ_EVENT) && $conv->get_profile_owner() == local_channel())
+		if((in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT])) && $conv->get_profile_owner() == local_channel())
 			$has_event = true;
 
 		$like = [];

Zotlabs/Lib/ThreadListener.php

@@ -4,18 +4,30 @@ namespace Zotlabs\Lib;
 
 class ThreadListener {
 
+	public static function isEnabled() {
+		return Config::Get('system','enable_thread_listener');
+	}
+
 	static public function store($target_id,$portable_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return true;
+		}
+
 		$x = self::fetch($target_id,$portable_id,$ltype = 0);
-		if(! $x) {  
+		if(! $x) {
 			$r = q("insert into listeners ( target_id, portable_id, ltype ) values ( '%s', '%s' , %d ) ",
 				dbesc($target_id),
 				dbesc($portable_id),
 				intval($ltype)
 			);
-		}  	
+		}
 	}
 
 	static public function fetch($target_id,$portable_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return false;
+		}
+
 		$x = q("select * from listeners where target_id = '%s' and portable_id = '%s' and ltype = %d limit 1",
 			dbesc($target_id),
 			dbesc($portable_id),
@@ -28,6 +40,10 @@ class ThreadListener {
 	}
 
 	static public function fetch_by_target($target_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return [];
+		}
+
 		$x = q("select * from listeners where target_id = '%s' and ltype = %d",
 			dbesc($target_id),
 			intval($ltype)

Zotlabs/Lib/Webfinger.php

@@ -52,15 +52,21 @@ class Webfinger {
 
 		if(strpos($resource,'http') === 0) {
 			$m = parse_url($resource);
-			if($m) {
-				if(isset($m['scheme']) && $m['scheme'] !== 'https') {
-					return false;
-				}
-				self::$server = $m['host'] . ((isset($m['port'])) ? ':' . $m['port'] : '');
-			}
-			else {
+
+			if (!$m) {
 				return false;
 			}
+
+			if(isset($m['scheme']) && $m['scheme'] !== 'https') {
+				return false;
+			}
+
+			if(!isset($m['host'])) {
+				return false;
+			}
+
+			self::$server = $m['host'] . ((isset($m['port'])) ? ':' . $m['port'] : '');
+
 		}
 		elseif(strpos($resource,'tag:') === 0) {
 			$arr = explode(':',$resource); // split the tag

Zotlabs/Module/Acl.php

@@ -378,6 +378,7 @@ class Acl extends \Zotlabs\Web\Controller {
 							"name"     => $g['name'],
 							"id"	   => urlencode($g['id']),
 							"xid"      => $g['hash'],
+							"url"      => $g['url'],
 							"link"     => $clink,
 							"nick"     => ((strpos($g['nick'],'@')) ? substr($g['nick'],0,strpos($g['nick'],'@')) : $g['nick']),
 							"self"     => (intval($g['abook_self']) ? 'abook-self' : ''),

Zotlabs/Module/Activity.php

@@ -182,7 +182,7 @@ class Activity extends Controller {
 				return;
 			}
 
-			$ob_authorise = false;
+			$ob_authorize = false;
 			$item_uid = 0;
 
 			$bear = ZlibActivity::token_from_request();

Zotlabs/Module/Admin/Queueworker.php

@@ -0,0 +1,121 @@
+<?php
+
+namespace Zotlabs\Module\Admin;
+
+use App;
+use Zotlabs\Web\Controller;
+
+class Queueworker extends Controller {
+
+	function init() {
+
+	}
+
+	function post() {
+
+		check_form_security_token('form_security_token', 'queueworker');
+
+		$maxqueueworkers = intval($_POST['queueworker_maxworkers']);
+		$maxqueueworkers = ($maxqueueworkers > 3) ? $maxqueueworkers : 4;
+		set_config('queueworker', 'max_queueworkers', $maxqueueworkers);
+
+		$maxworkerage = intval($_POST['queueworker_max_age']);
+		$maxworkerage = ($maxworkerage >= 120) ? $maxworkerage : 300;
+		set_config('queueworker', 'queueworker_max_age', $maxworkerage);
+
+		$queueworkersleep = intval($_POST['queue_worker_sleep']);
+		$queueworkersleep = ($queueworkersleep > 100) ? $queueworkersleep : 100;
+		set_config('queueworker', 'queue_worker_sleep', $queueworkersleep);
+
+		$auto_queue_worker_sleep = intval($_POST['auto_queue_worker_sleep']);
+		set_config('queueworker', 'auto_queue_worker_sleep', $auto_queue_worker_sleep);
+
+		goaway(z_root() . '/admin/queueworker');
+	}
+
+	function get() {
+
+		$content = "<H1>Queue Status</H1>\n";
+
+		$r = q('select count(*) as total from workerq');
+
+		$content .= "<H4>There are " . $r[0]['total'] . " queue items to be processed.</H4>";
+
+		$r = dbq("select count(*) as qworkers from workerq where workerq_reservationid is not null");
+
+		$content .= "<H4>Active workers: " . $r[0]['qworkers'] . "</H4>";
+
+		$r = dbq("select workerq_cmd, count(*) as total from workerq where true group by workerq_cmd");
+
+		if ($r) {
+			$content .= "<H4>Work items</H4>";
+			foreach($r as $rr) {
+				$content .= $rr['workerq_cmd'] . ': ' . $rr['total'] . '<br>';
+			}
+		}
+
+		$maxqueueworkers = get_config('queueworker', 'max_queueworkers', 4);
+		$maxqueueworkers = ($maxqueueworkers > 3) ? $maxqueueworkers : 4;
+
+		$sc = '';
+
+		$sc .= replace_macros(get_markup_template('field_input.tpl'), [
+			'$field' => [
+				'queueworker_maxworkers',
+				t('Max queueworker threads'),
+				$maxqueueworkers,
+				t('Minimum 4, default 4')
+			]
+		]);
+
+		$workermaxage = get_config('queueworker', 'queueworker_max_age');
+		$workermaxage = ($workermaxage >= 120) ? $workermaxage : 300;
+
+		$sc .= replace_macros(get_markup_template('field_input.tpl'), [
+			'$field' => [
+				'queueworker_max_age',
+				t('Assume workers dead after'),
+				$workermaxage,
+				t('Minimum 120, default 300 seconds')
+			]
+		]);
+
+		$queueworkersleep = get_config('queueworker', 'queue_worker_sleep');
+		$queueworkersleep = ($queueworkersleep > 100) ? $queueworkersleep : 100;
+
+		$auto_queue_worker_sleep = get_config('queueworker', 'auto_queue_worker_sleep', 0);
+
+		$sc .= replace_macros(get_markup_template('field_input.tpl'), [
+			'$field' => [
+				'queue_worker_sleep',
+				t('Pause before starting next task'),
+				$queueworkersleep,
+				t('Minimum 100, default 100 microseconds'),
+				'',
+				(($auto_queue_worker_sleep) ? 'disabled' : '')
+			]
+		]);
+
+		$sc .= replace_macros(get_markup_template('field_checkbox.tpl'), [
+			'$field' => [
+				'auto_queue_worker_sleep',
+				t('Automatically adjust pause before starting next task'),
+				$auto_queue_worker_sleep,
+			]
+		]);
+
+		$tpl = get_markup_template('settings_addon.tpl');
+		$content .= replace_macros($tpl, [
+				'$action_url' => 'admin/queueworker',
+				'$form_security_token' => get_form_security_token('queueworker'),
+				'$title' => t('Queueworker Settings'),
+				'$content' => $sc,
+				'$baseurl' => z_root(),
+				'$submit' => t('Save')
+			]
+		);
+
+		return $content;
+
+	}
+}

Zotlabs/Module/Admin/Security.php

@@ -7,12 +7,12 @@ class Security {
 
 	function post() {
 		check_form_security_token_redirectOnErr('/admin/security', 'admin_security');
-	
+
 		$allowed_email        = ((x($_POST,'allowed_email'))	    ? notags(trim($_POST['allowed_email']))		: '');
 		$not_allowed_email    = ((x($_POST,'not_allowed_email'))	? notags(trim($_POST['not_allowed_email']))		: '');
 
 		set_config('system','allowed_email', $allowed_email);
-		set_config('system','not_allowed_email', $not_allowed_email);	
+		set_config('system','not_allowed_email', $not_allowed_email);
 
 		$block_public         = ((x($_POST,'block_public'))		? True	: False);
 		set_config('system','block_public',$block_public);
@@ -25,22 +25,22 @@ class Security {
 
 		$ws = $this->trim_array_elems(explode("\n",$_POST['whitelisted_sites']));
 		set_config('system','whitelisted_sites',$ws);
-	
+
 		$bs = $this->trim_array_elems(explode("\n",$_POST['blacklisted_sites']));
 		set_config('system','blacklisted_sites',$bs);
-	
+
 		$wc = $this->trim_array_elems(explode("\n",$_POST['whitelisted_channels']));
 		set_config('system','whitelisted_channels',$wc);
-	
+
 		$bc = $this->trim_array_elems(explode("\n",$_POST['blacklisted_channels']));
 		set_config('system','blacklisted_channels',$bc);
-	
+
 		$embed_sslonly         = ((x($_POST,'embed_sslonly'))		? True	: False);
 		set_config('system','embed_sslonly',$embed_sslonly);
-	
+
 		$we = $this->trim_array_elems(explode("\n",$_POST['embed_allow']));
 		set_config('system','embed_allow',$we);
-	
+
 		$be = $this->trim_array_elems(explode("\n",$_POST['embed_deny']));
 		set_config('system','embed_deny',$be);
 
@@ -49,47 +49,54 @@ class Security {
 
 		$inline_pdf = ((x($_POST,'inline_pdf')) ? intval($_POST['inline_pdf']) : 0);
 		set_config('system', 'inline_pdf' , $inline_pdf);
-	
+
 		$ts = ((x($_POST,'transport_security')) ? True : False);
 		set_config('system','transport_security_header',$ts);
 
 		$cs = ((x($_POST,'content_security')) ? True : False);
 		set_config('system','content_security_policy',$cs);
 
+		$trusted_directory_servers = $this->trim_array_elems(explode("\n", $_POST['trusted_directory_servers']));
+		set_config('system', 'trusted_directory_servers', $trusted_directory_servers);
+
 		goaway(z_root() . '/admin/security');
 	}
-	
-	
+
+
 
 	function get() {
-	
+
 		$whitesites = get_config('system','whitelisted_sites');
 		$whitesites_str = ((is_array($whitesites)) ? implode("\n",$whitesites) : '');
-	
+
 		$blacksites = get_config('system','blacklisted_sites');
 		$blacksites_str = ((is_array($blacksites)) ? implode("\n",$blacksites) : '');
-	
-	
+
+
 		$whitechannels = get_config('system','whitelisted_channels');
 		$whitechannels_str = ((is_array($whitechannels)) ? implode("\n",$whitechannels) : '');
-	
+
 		$blackchannels = get_config('system','blacklisted_channels');
 		$blackchannels_str = ((is_array($blackchannels)) ? implode("\n",$blackchannels) : '');
-	
-	
+
+
 		$whiteembeds = get_config('system','embed_allow');
 		$whiteembeds_str = ((is_array($whiteembeds)) ? implode("\n",$whiteembeds) : '');
-	
+
 		$blackembeds = get_config('system','embed_deny');
 		$blackembeds_str = ((is_array($blackembeds)) ? implode("\n",$blackembeds) : '');
-	
+
+		$trusted_directory_servers = get_config('system', 'trusted_directory_servers');
+		$trusted_directory_servers_str = ((is_array($trusted_directory_servers)) ? implode("\n", $trusted_directory_servers) : '');
+		$is_dir = (intval(get_config('system', 'directory_mode', DIRECTORY_MODE_NORMAL)) !== DIRECTORY_MODE_NORMAL);
+
 		$embed_coop = intval(get_config('system','embed_coop'));
-	
+
 		if((! $whiteembeds) && (! $blackembeds)) {
 			$embedhelp1 = t("By default, unfiltered HTML is allowed in embedded media. This is inherently insecure.");
 		}
 
-		$embedhelp2 = t("The recommended setting is to only allow unfiltered HTML from the following sites:"); 
+		$embedhelp2 = t("The recommended setting is to only allow unfiltered HTML from the following sites:");
 		$embedhelp3 = t("https://youtube.com/<br />https://www.youtube.com/<br />https://youtu.be/<br />https://vimeo.com/<br />https://soundcloud.com/<br />");
 		$embedhelp4 = t("All other embedded content will be filtered, <strong>unless</strong> embedded content from that site is explicitly blocked.");
 
@@ -99,7 +106,7 @@ class Security {
 			'$page' => t('Security'),
 			'$form_security_token' => get_form_security_token('admin_security'),
 	        '$block_public'     => array('block_public', t("Block public"), get_config('system','block_public'), t("Check to block public access to all otherwise public personal pages on this site unless you are currently authenticated.")),
-			'$cloud_noroot'     => [ 'cloud_noroot', t('Provide a cloud root directory'), 1 - intval(get_config('system','cloud_disable_siteroot')), t('The cloud root directory lists all channel names which provide public files') ], 
+			'$cloud_noroot'     => [ 'cloud_noroot', t('Provide a cloud root directory'), 1 - intval(get_config('system','cloud_disable_siteroot')), t('The cloud root directory lists all channel names which provide public files') ],
 			'$cloud_disksize'     => [ 'cloud_disksize', t('Show total disk space available to cloud uploads'), intval(get_config('system','cloud_report_disksize')), '' ],
 			'$transport_security' => array('transport_security', t('Set "Transport Security" HTTP header'),intval(get_config('system','transport_security_header')),''),
 			'$content_security' => array('content_security', t('Set "Content Security Policy" HTTP header'),intval(get_config('system','content_security_policy')),''),
@@ -115,6 +122,8 @@ class Security {
 			'$thumbnail_security'   => [ 'thumbnail_security', t("Allow SVG thumbnails in file browser"), get_config('system','thumbnail_security',0), t("WARNING: SVG images may contain malicious code.") ],
 			'$inline_pdf'   => [ 'inline_pdf', t("Allow embedded (inline) PDF files"), get_config('system','inline_pdf',0), '' ],
 
+			'$trusted_directory_servers' => (($is_dir) ? ['trusted_directory_servers', t('Additional trusted directory server URLs'), $trusted_directory_servers_str, t('Accept directory flags (spam, nsfw) from those servers. One per line like https://example.tld')] : ''),
+
 //	        '$embed_coop'     => array('embed_coop', t('Cooperative embed security'), $embed_coop, t('Enable to share embed security with other compatible sites/hubs')),
 
 			'$submit' => t('Submit')
@@ -124,7 +133,7 @@ class Security {
 
 	function trim_array_elems($arr) {
 		$narr = array();
-	
+
 		if($arr && is_array($arr)) {
 			for($x = 0; $x < count($arr); $x ++) {
 				$y = trim($arr[$x]);
@@ -134,6 +143,6 @@ class Security {
 		}
 		return $narr;
 	}
-	
-	
+
+
 }

Zotlabs/Module/Admin/Site.php

@@ -67,7 +67,6 @@ class Site {
 		$open_pubstream     =   ((x($_POST,'open_pubstream')) ? True : False);
 		$login_on_homepage	=	((x($_POST,'login_on_homepage'))		? True	:	False);
 		$enable_context_help = ((x($_POST,'enable_context_help'))		? True	:	False);
-		$global_directory     = ((x($_POST,'directory_submit_url'))	? notags(trim($_POST['directory_submit_url']))	: '');
 		$no_community_page    = !((x($_POST,'no_community_page'))	? True	:	False);
 		$default_expire_days  = ((array_key_exists('default_expire_days',$_POST)) ? intval($_POST['default_expire_days']) : 0);
 		$active_expire_days  = ((array_key_exists('active_expire_days',$_POST)) ? intval($_POST['active_expire_days']) : 7);
@@ -100,7 +99,7 @@ class Site {
 		$reg_expire 	   = (preg_match('/^[a-z]{1,1}$/', $regexpireu) ? $regexpiren . $regexpireu : '');
 
 		$imagick_path      = ((x($_POST,'imagick_path'))     ? trim($_POST['imagick_path'])   : '');
-		$force_queue       = ((intval($_POST['force_queue']) > 0) ? intval($_POST['force_queue'])   : 3000);
+		//$force_queue       = ((intval($_POST['force_queue']) > 0) ? intval($_POST['force_queue'])   : 3000);
 		$pub_incl = escape_tags(trim($_POST['pub_incl']));
 		$pub_excl = escape_tags(trim($_POST['pub_excl']));
 
@@ -181,7 +180,7 @@ class Site {
 			set_config('system', 'admininfo', $admininfo);
 		}
 		set_config('system','siteinfo',$siteinfo);
-		set_config('system', 'language', $language);
+		//set_config('system', 'language', $language);
 		set_config('system', 'theme', $theme);
 		//		if ( $theme_mobile === '---' ) {
 		//			del_config('system', 'mobile_theme');
@@ -205,12 +204,7 @@ class Site {
 		set_config('system','disable_discover_tab', $disable_discover_tab);
 		set_config('system','site_firehose', $site_firehose);
 		set_config('system','open_pubstream', $open_pubstream);
-		set_config('system','force_queue_threshold', $force_queue);
-		if ($global_directory == '') {
-			del_config('system', 'directory_submit_url');
-		} else {
-			set_config('system', 'directory_submit_url', $global_directory);
-		}
+		//set_config('system','force_queue_threshold', $force_queue);
 
 		set_config('system','no_community_page', $no_community_page);
 		set_config('system','no_utf', $no_utf);
@@ -283,7 +277,7 @@ class Site {
 		}
 
 		$dir_choices = null;
-		$dirmode = get_config('system','directory_mode');
+		$dirmode = get_config('system', 'directory_mode', DIRECTORY_MODE_NORMAL);
 		$realm = get_directory_realm();
 
 		// directory server should not be set or settable unless we are a directory client
@@ -301,6 +295,12 @@ class Site {
 					$dir_choices[$xx['site_url']] = $xx['site_url'];
 				}
 			}
+			if ($realm === DIRECTORY_REALM) {
+				$fallback_servers = get_directory_fallback_servers();
+				foreach ($fallback_servers as $fallback_server) {
+					$dir_choices[$fallback_server] = $fallback_server;
+				}
+			}
 		}
 
 		/* Banner */
@@ -425,7 +425,7 @@ class Site {
 			'$banner'			=> array('banner', t("Banner/Logo"), $banner, t('Unfiltered HTML/CSS/JS is allowed')),
 			'$admininfo'		=> array('admininfo', t("Administrator Information"), $admininfo, t("Contact information for site administrators.  Displayed on siteinfo page.  BBCode can be used here")),
 			'$siteinfo'		=> array('siteinfo', t('Site Information'), get_config('system','siteinfo'), t("Publicly visible description of this site.  Displayed on siteinfo page.  BBCode can be used here")),
-			'$language' 		=> array('language', t("System language"), get_config('system','language'), "", $lang_choices),
+			//'$language' 		=> array('language', t("System language"), get_config('system','language'), "", $lang_choices),
 			'$theme' 			=> array('theme', t("System theme"), get_config('system','theme'), t("Default system theme - may be over-ridden by user profiles - <a href='#' id='cnftheme'>change theme settings</a>"), $theme_choices),
 		//			'$theme_mobile' 	=> array('theme_mobile', t("Mobile system theme"), get_config('system','mobile_theme'), t("Theme for mobile devices"), $theme_choices_mobile),
 		//			'$site_channel' 	=> array('site_channel', t("Channel to use for this website's static pages"), get_config('system','site_channel'), t("Site Channel")),
@@ -508,10 +508,10 @@ class Site {
 			'$frontpage'	=> array('frontpage', t("Site homepage to show visitors (default: login box)"), get_config('system','frontpage'), t("example: 'pubstream' to show public stream, 'page/sys/home' to show a system webpage called 'home' or 'include:home.html' to include a file.")),
 			'$mirror_frontpage'	=> array('mirror_frontpage', t("Preserve site homepage URL"), get_config('system','mirror_frontpage'), t('Present the site homepage in a frame at the original location instead of redirecting')),
 			'$allowed_sites'	=> array('allowed_sites', t("Allowed friend domains"), get_config('system','allowed_sites'), t("Comma separated list of domains which are allowed to establish friendships with this site. Wildcards are accepted. Empty to allow any domains")),
-			'$force_publish'	=> array('publish_all', t("Force publish"), get_config('system','publish_all'), t("Check to force all profiles on this site to be listed in the site directory.")),
-			'$disable_discover_tab'	=> array('disable_discover_tab', t('Import Public Streams'), $discover_tab, t('Import and allow access to public content pulled from other sites. Warning: this content is unmoderated.')),
-			'$site_firehose'	=> array('site_firehose', t('Site only Public Streams'), get_config('system','site_firehose'), t('Allow access to public content originating only from this site if Imported Public Streams are disabled.')),
-			'$open_pubstream'	=> array('open_pubstream', t('Allow anybody on the internet to access the Public streams'), get_config('system','open_pubstream',1), t('Disable to require authentication before viewing. Warning: this content is unmoderated.')),
+			'$force_publish'	=> array('publish_all', t("Force publish"), get_config('system','publish_all'), t("Check to force all profiles on this site to be listed in the site directory")),
+			'$disable_discover_tab'	=> array('disable_discover_tab', t('Enable public stream'), $discover_tab, t('Enable the public stream. Warning: this content is unmoderated')),
+			'$site_firehose'	=> array('site_firehose', t('Site only public stream'), get_config('system','site_firehose'), t('Restrict the public stream to content originating at this site')),
+			'$open_pubstream'	=> array('open_pubstream', t('Allow anybody on the internet to access the public streams'), get_config('system','open_pubstream',1), t('Disable to require authentication before viewing')),
 			'$incl'           => array('pub_incl',t('Only import Public stream posts with this text'), get_config('system','pubstream_incl'),t('words one per line or #tags or /patterns/ or lang=xx, leave blank to import all posts')),
 			'$excl'           => array('pub_excl',t('Do not import Public stream posts with this text'), get_config('system','pubstream_excl'),t('words one per line or #tags or /patterns/ or lang=xx, leave blank to import all posts')),
 
@@ -532,7 +532,7 @@ class Site {
 			'$timeout'			=> array('timeout', t("Network timeout"), (x(get_config('system','curl_timeout'))?get_config('system','curl_timeout'):60), t("Value is in seconds. Set to 0 for unlimited (not recommended).")),
 			'$delivery_interval'			=> array('delivery_interval', t("Delivery interval"), (x(get_config('system','delivery_interval'))?get_config('system','delivery_interval'):2), t("Delay background delivery processes by this many seconds to reduce system load. Recommend: 4-5 for shared hosts, 2-3 for virtual private servers. 0-1 for large dedicated servers.")),
 			'$delivery_batch_count' => array('delivery_batch_count', t('Deliveries per process'),(x(get_config('system','delivery_batch_count'))?get_config('system','delivery_batch_count'):1), t("Number of deliveries to attempt in a single operating system process. Adjust if necessary to tune system performance. Recommend: 1-5.")),
-			'$force_queue'			=> array('force_queue', t("Queue Threshold"), get_config('system','force_queue_threshold',3000), t("Always defer immediate delivery if queue contains more than this number of entries.")),
+			//'$force_queue'			=> array('force_queue', t("Queue Threshold"), get_config('system','force_queue_threshold',3000), t("Always defer immediate delivery if queue contains more than this number of entries.")),
 			'$poll_interval'			=> array('poll_interval', t("Poll interval"), (x(get_config('system','poll_interval'))?get_config('system','poll_interval'):2), t("Delay background polling processes by this many seconds to reduce system load. If 0, use delivery interval.")),
 			'$imagick_path'			=> array('imagick_path', t("Path to ImageMagick convert program"), get_config('system','imagick_convert_path'), t("If set, use this program to generate photo thumbnails for huge images ( > 4000 pixels in either dimension), otherwise memory exhaustion may occur. Example: /usr/bin/convert")),
 			'$maxloadavg'			=> array('maxloadavg', t("Maximum Load Average"), ((intval(get_config('system','maxloadavg')) > 0)?get_config('system','maxloadavg'):50), t("Maximum system load before delivery and poll processes are deferred - default 50.")),

Zotlabs/Module/Cal.php

@@ -65,13 +65,9 @@ class Cal extends Controller {
 
 		nav_set_selected('Calendar');
 
-		head_add_css('/library/fullcalendar/packages/core/main.min.css');
-		head_add_css('/library/fullcalendar/packages/daygrid/main.min.css');
+		head_add_js('/library/fullcalendar/dist/index.global.js');
 		head_add_css('cdav_calendar.css');
 
-		head_add_js('/library/fullcalendar/packages/core/main.min.js');
-		head_add_js('/library/fullcalendar/packages/daygrid/main.min.js');
-
 		$sql_extra = permissions_sql($channel['channel_id'], get_observer_hash(), 'event');
 
 		if(! perm_is_allowed($channel['channel_id'], get_observer_hash(), 'view_contacts') || App::$profile['hide_friends'])

Zotlabs/Module/Cdav.php

@@ -332,9 +332,9 @@ class Cdav extends Controller {
 				} while ($duplicate == true);
 
 				$properties = [
-					'{DAV:}displayname' => $_REQUEST['{DAV:}displayname'],
-					'{http://apple.com/ns/ical/}calendar-color' => $_REQUEST['color'],
-					'{urn:ietf:params:xml:ns:caldav}calendar-description' => $channel['channel_name']
+					'{DAV:}displayname' => escape_tags($_REQUEST['{DAV:}displayname']),
+					'{http://apple.com/ns/ical/}calendar-color' => escape_tags($_REQUEST['color']),
+					'{urn:ietf:params:xml:ns:caldav}calendar-description' => escape_tags($channel['channel_name'])
 				];
 
 				$id = $caldavBackend->createCalendar($principalUri, $calendarUri, $properties);
@@ -366,7 +366,7 @@ class Cdav extends Controller {
 
 				$allday = $_REQUEST['allday'];
 
-				$title = $_REQUEST['title'];
+				$title = escape_tags($_REQUEST['title']);
 				$start = datetime_convert('UTC', 'UTC', $_REQUEST['dtstart']);
 				$dtstart = new \DateTime($start);
 
@@ -374,8 +374,8 @@ class Cdav extends Controller {
 					$end = datetime_convert('UTC', 'UTC', $_REQUEST['dtend']);
 					$dtend = new \DateTime($end);
 				}
-				$description = $_REQUEST['description'];
-				$location = $_REQUEST['location'];
+				$description = escape_tags($_REQUEST['description']);
+				$location = escape_tags($_REQUEST['location']);
 
 				do {
 					$duplicate = false;
@@ -441,8 +441,8 @@ class Cdav extends Controller {
 				$cdavdata = $this->get_cdav_data($id[0], 'calendarinstances');
 
 				$mutations = [
-					'{DAV:}displayname' => $_REQUEST['{DAV:}displayname'],
-					'{http://apple.com/ns/ical/}calendar-color' => $_REQUEST['color']
+					'{DAV:}displayname' => escape_tags($_REQUEST['{DAV:}displayname']),
+					'{http://apple.com/ns/ical/}calendar-color' => escape_tags($_REQUEST['color'])
 				];
 
 				$patch = new \Sabre\DAV\PropPatch($mutations);
@@ -471,18 +471,18 @@ class Cdav extends Controller {
 				$timezone = ((x($_POST,'timezone_select')) ? escape_tags(trim($_POST['timezone_select'])) : '');
 				$tz = (($timezone) ? $timezone : date_default_timezone_get());
 
-				$allday = $_REQUEST['allday'];
+				$allday = intval($_REQUEST['allday']);
 
-				$uri = $_REQUEST['uri'];
-				$title = $_REQUEST['title'];
+				$uri = escape_tags($_REQUEST['uri']);
+				$title = escape_tags($_REQUEST['title']);
 				$start = datetime_convert('UTC', 'UTC', $_REQUEST['dtstart']);
 				$dtstart = new \DateTime($start);
 				if($_REQUEST['dtend']) {
 					$end = datetime_convert('UTC', 'UTC', $_REQUEST['dtend']);
 					$dtend = new \DateTime($end);
 				}
-				$description = $_REQUEST['description'];
-				$location = $_REQUEST['location'];
+				$description = escape_tags($_REQUEST['description']);
+				$location = escape_tags($_REQUEST['location']);
 
 				$object = $caldavBackend->getCalendarObject($id, $uri);
 
@@ -654,7 +654,7 @@ class Cdav extends Controller {
 						$duplicate = true;
 				} while ($duplicate == true);
 
-				$properties = ['{DAV:}displayname' => $_REQUEST['{DAV:}displayname']];
+				$properties = ['{DAV:}displayname' => escape_tags($_REQUEST['{DAV:}displayname'])];
 
 				$carddavBackend->createAddressBook($principalUri, $addressbookUri, $properties);
 
@@ -668,9 +668,9 @@ class Cdav extends Controller {
 			}
 
 			//edit addressbook
-			if($_REQUEST['{DAV:}displayname'] && $_REQUEST['edit'] && intval($_REQUEST['id'])) {
+			if($_REQUEST['{DAV:}displayname'] && $_REQUEST['edit'] && $_REQUEST['id']) {
 
-				$id = $_REQUEST['id'];
+				$id = intval($_REQUEST['id']);
 
 				if(! cdav_perms($id,$addressbooks))
 					return;
@@ -678,7 +678,7 @@ class Cdav extends Controller {
 				$cdavdata = $this->get_cdav_data($id, 'addressbooks');
 
 				$mutations = [
-					'{DAV:}displayname' => $_REQUEST['{DAV:}displayname']
+					'{DAV:}displayname' => escape_tags($_REQUEST['{DAV:}displayname'])
 				];
 
 				$patch = new \Sabre\DAV\PropPatch($mutations);
@@ -901,19 +901,9 @@ class Cdav extends Controller {
 
 		//Display calendar(s) here
 		if(argc() <= 3 && argv(1) === 'calendar') {
-
-			head_add_css('/library/fullcalendar/packages/core/main.min.css');
-			head_add_css('/library/fullcalendar/packages/daygrid/main.min.css');
-			head_add_css('/library/fullcalendar/packages/timegrid/main.min.css');
-			head_add_css('/library/fullcalendar/packages/list/main.min.css');
+			head_add_js('/library/fullcalendar/dist/index.global.js');
 			head_add_css('cdav_calendar.css');
 
-			head_add_js('/library/fullcalendar/packages/core/main.min.js');
-			head_add_js('/library/fullcalendar/packages/interaction/main.min.js');
-			head_add_js('/library/fullcalendar/packages/daygrid/main.min.js');
-			head_add_js('/library/fullcalendar/packages/timegrid/main.min.js');
-			head_add_js('/library/fullcalendar/packages/list/main.min.js');
-
 			$o = '';
 			$sources = '';
 			$resource_id = '';

Zotlabs/Module/Channel.php

@@ -247,9 +247,7 @@ class Channel extends Controller {
 				$channel_acl = ['allow_cid' => '', 'allow_gid' => '', 'deny_cid' => '', 'deny_gid' => ''];
 			}
 
-
 			if ($perms['post_wall']) {
-
 				$x = [
 					'is_owner'            => $is_owner,
 					'allow_location'      => ((($is_owner || $observer) && (intval(get_pconfig(App::$profile['profile_uid'], 'system', 'use_browser_location')))) ? true : false),
@@ -273,6 +271,12 @@ class Channel extends Controller {
 				$o .= status_editor($a, $x, false, 'Channel');
 			}
 
+			// Add pinned content
+			if (!x($_REQUEST, 'mid') && !$search) {
+				$pinned = new \Zotlabs\Widget\Pinned;
+				$r = $pinned->widget(intval(App::$profile['profile_uid']), [ITEM_TYPE_POST]);
+				$o .= $r['html'];
+			}
 		}
 
 
@@ -423,17 +427,14 @@ class Channel extends Controller {
 			$items = [];
 		}
 
-		// Add pinned content
-		if (!x($_REQUEST, 'mid') && !$search) {
-			$pinned = new \Zotlabs\Widget\Pinned;
-			$r      = $pinned->widget(intval(App::$profile['profile_uid']), [ITEM_TYPE_POST]);
-			$o      .= $r['html'];
-		}
+
 
 		$mode = (($search) ? 'search' : 'channel');
 
 		if ((!$update) && (!$load)) {
 
+
+
 			//if we got a decoded hash we must encode it again before handing to javascript
 			$mid = gen_link_id($mid);
 

Zotlabs/Module/Chanview.php

@@ -16,17 +16,17 @@ class Chanview extends \Zotlabs\Web\Controller {
 
 		$r = null;
 
-		if(isset($_REQUEST['hash']) && $_REQUEST['hash']) {
+		if(!empty($_REQUEST['hash'])) {
 			$r = q("select * from xchan where xchan_hash = '%s' and xchan_deleted = 0",
 				dbesc($_REQUEST['hash'])
 			);
 		}
-		if(isset($_REQUEST['address']) && $_REQUEST['address']) {
+		if(!empty($_REQUEST['address'])) {
 			$r = q("select * from xchan where xchan_addr = '%s' and xchan_deleted = 0",
 				dbesc(punify($_REQUEST['address']))
 			);
 		}
-		elseif(local_channel() && isset($_REQUEST['cid']) && $_REQUEST['cid']) {
+		elseif(local_channel() && !empty($_REQUEST['cid'])) {
 			$r = q("SELECT abook.*, xchan.*
 				FROM abook left join xchan on abook_xchan = xchan_hash
 				WHERE abook_channel = %d and abook_id = %d and xchan_deleted = 0",
@@ -34,8 +34,7 @@ class Chanview extends \Zotlabs\Web\Controller {
 				intval($_REQUEST['cid'])
 			);
 		}
-		elseif(isset($_REQUEST['url']) && $_REQUEST['url']) {
-
+		elseif(!empty($_REQUEST['url'])) {
 			// if somebody re-installed they will have more than one xchan, use the most recent name date as this is
 			// the most useful consistently ascending table item we have.
 
@@ -56,7 +55,7 @@ class Chanview extends \Zotlabs\Web\Controller {
 		if(! App::$poi) {
 			logger('mod_chanview: fallback');
 
-			if(isset($_REQUEST['address']) && $_REQUEST['address']) {
+			if(!empty($_REQUEST['address'])) {
 				$href = Webfinger::zot_url(punify($_REQUEST['address']));
 				if($href) {
 					$_REQUEST['url'] = $href;
@@ -65,9 +64,9 @@ class Chanview extends \Zotlabs\Web\Controller {
 
 			$r = null;
 
-			if(isset($_REQUEST['url']) && $_REQUEST['url']) {
+			if(!empty($_REQUEST['url'])) {
 
-				$zf = Zotfinger::exec($_REQUEST['url'], null);
+				$zf = Zotfinger::exec($_REQUEST['url']);
 
 				if(array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $_REQUEST['url'] && intval($zf['signature']['header_valid'])) {
 					Libzot::import_xchan($zf['data']);
@@ -79,6 +78,7 @@ class Chanview extends \Zotlabs\Web\Controller {
 					}
 				}
 				if(! $r) {
+
 					if(discover_by_webbie($_REQUEST['url'])) {
 						$r = q("select * from xchan where xchan_url = '%s' and xchan_deleted = 0",
 							dbesc($_REQUEST['url'])

Zotlabs/Module/Cloud.php

@@ -100,7 +100,6 @@ class Cloud extends Controller {
 
 
 		// over-ride the default XML output on thrown exceptions
-
 		$server->on('exception', [ $this, 'DAVException' ]);
 
 		// All we need to do now, is to fire up the server
@@ -117,21 +116,19 @@ class Cloud extends Controller {
 	function DAVException($err) {
 
 		if($err instanceof \Sabre\DAV\Exception\NotFound) {
-			notice( t('Not found') . EOL);
+			\App::$page['content'] = '<h2>404 Not found</h2>';
 		}
 		elseif($err instanceof \Sabre\DAV\Exception\Forbidden) {
-			notice( t('Permission denied') . EOL);
+			\App::$page['content'] = '<h2>403 Forbidden</h2>';
 		}
 		elseif($err instanceof \Sabre\DAV\Exception\NotImplemented) {
-			// notice( t('Please refresh page') . EOL);
 			goaway(z_root() . '/' . \App::$query_string);
 		}
 		else {
-			notice( t('Unknown error') . EOL);
+			\App::$page['content'] = '<h2>Unknown error</h2>';
 		}
 
 		construct_page();
-
 		killme();
 	}
 

Zotlabs/Module/Connections.php

@@ -145,6 +145,7 @@ class Connections extends \Zotlabs\Web\Controller {
 		}
 
 		$search = ((x($_REQUEST,'search')) ? notags(trim($_REQUEST['search'])) : '');
+		$search_xchan = ((x($_REQUEST,'search_xchan')) ? notags(trim($_REQUEST['search_xchan'])) : '');
 
 		$tabs = array(
 			/*
@@ -233,10 +234,15 @@ class Connections extends \Zotlabs\Web\Controller {
 
 		if($search) {
 			$search_hdr = $search;
-			$search_txt = dbesc(protect_sprintf(preg_quote($search)));
-			$searching = true;
+			$search_txt = (($search_xchan) ? urldecode($search_xchan) : preg_quote($search));
+
+			if ($search_xchan) {
+				$sql_extra .= " AND xchan_hash = '" . protect_sprintf(dbesc($search_txt)) . "' ";
+			}
+			else {
+				$sql_extra .= " AND xchan_name LIKE '%%" . protect_sprintf(dbesc($search_txt)) . "%%' ";
+			}
 		}
-		$sql_extra .= (($searching) ? protect_sprintf(" AND xchan_name like '%$search_txt%' ") : "");
 
 		if(isset($_REQUEST['gid']) && $_REQUEST['gid']) {
 			$sql_extra .= " and xchan_hash in ( select xchan from pgrp_member where gid = " . intval($_REQUEST['gid']) . " and uid = " . intval(local_channel()) . " ) ";
@@ -396,7 +402,7 @@ class Connections extends \Zotlabs\Web\Controller {
 				'$search' => $search_hdr,
 				'$label' => t('Search'),
 				'$role_label' => t('Contact role'),
-				'$desc' => t('Search your connections'),
+				'$desc' => $search ?? t('Search your connections'),
 				'$finding' => (($searching) ? t('Contact search') . ": '" . $search . "'" : ""),
 				'$submit' => t('Find'),
 				'$edit' => t('Edit'),

Zotlabs/Module/Conversation.php

@@ -0,0 +1,188 @@
+<?php
+
+namespace Zotlabs\Module;
+
+use Zotlabs\Web\Controller;
+use Zotlabs\Lib\ActivityStreams;
+use Zotlabs\Lib\Activity;
+use Zotlabs\Lib\Libzot;
+use Zotlabs\Web\HTTPSig;
+use Zotlabs\Lib\LDSignatures;
+use Zotlabs\Lib\ThreadListener;
+use Zotlabs\Lib\Config;
+use App;
+
+class Conversation extends Controller {
+
+	public function init() {
+
+		if (ActivityStreams::is_as_request()) {
+			$item_id = argv(1);
+
+			if (!$item_id) {
+				http_status_exit(404, 'Not found');
+			}
+
+			$portable_id = EMPTY_STR;
+
+			$item_normal_extra = sprintf(" and not verb in ('%s', '%s') ",
+				dbesc(ACTIVITY_FOLLOW),
+				dbesc(ACTIVITY_UNFOLLOW)
+			);
+
+			$item_normal = " and item.item_hidden = 0 and item.item_type = 0 and item.item_unpublished = 0 and item.item_delayed = 0 and item.item_blocked = 0 $item_normal_extra ";
+
+			$i = null;
+
+			// do we have the item (at all)?
+
+			$r = q("select parent_mid from item where mid = '%s' or uuid = '%s' $item_normal order by item_wall desc limit 1",
+				dbesc(z_root() . '/item/' . $item_id),
+				dbesc($item_id)
+			);
+
+			if (!$r) {
+				http_status_exit(404, 'Not found');
+			}
+
+			// process an authenticated fetch
+
+			$sigdata = HTTPSig::verify(EMPTY_STR);
+			if ($sigdata['portable_id'] && $sigdata['header_valid']) {
+				$portable_id = $sigdata['portable_id'];
+				observer_auth($portable_id);
+
+				// first see if we have a copy of this item's parent owned by the current signer
+				// include xchans for all zot-like networks - these will have the same guid and public key
+
+				$x = q(
+					"select * from xchan where xchan_hash = '%s'",
+					dbesc($sigdata['portable_id'])
+				);
+
+				if ($x) {
+					$xchans = q(
+						"select xchan_hash from xchan where xchan_hash = '%s' OR ( xchan_guid = '%s' AND xchan_pubkey = '%s' ) ",
+						dbesc($sigdata['portable_id']),
+						dbesc($x[0]['xchan_guid']),
+						dbesc($x[0]['xchan_pubkey'])
+					);
+
+					if ($xchans) {
+						$hashes = ids_to_querystr($xchans, 'xchan_hash', true);
+						$i = q(
+							"select id as item_id from item where mid = '%s' $item_normal and owner_xchan in ( " . protect_sprintf($hashes) . " ) limit 1",
+							dbesc($r[0]['parent_mid'])
+						);
+					}
+				}
+			}
+			elseif (Config::get('system', 'require_authenticated_fetch', false)) {
+				http_status_exit(403, 'Permission denied');
+			}
+
+			// if we don't have a parent id belonging to the signer see if we can obtain one as a visitor that we have permission to access
+			// with a bias towards those items owned by channels on this site (item_wall = 1)
+
+			$sql_extra = item_permissions_sql(0);
+
+			if (!$i) {
+				$i = q("select id as item_id from item where mid = '%s' $item_normal $sql_extra order by item_wall desc limit 1",
+					dbesc($r[0]['parent_mid'])
+				);
+			}
+
+			if (!$i) {
+				http_status_exit(403, 'Forbidden');
+			}
+
+			$parents_str = ids_to_querystr($i, 'item_id');
+			$itemspage = 30;
+			$page = $_REQUEST['page'] ?? 1;
+			$offset = (intval($page) - 1) * $itemspage;
+
+			$items = dbq("SELECT item.*,
+				item.id AS item_id,
+				(SELECT count(*) FROM item WHERE item.parent IN ( $parents_str ) $item_normal) AS total
+				FROM item WHERE item.parent IN ( $parents_str ) $item_normal order by item.id LIMIT $itemspage OFFSET $offset"
+			);
+
+			if (!$items) {
+				http_status_exit(404, 'Not found');
+			}
+
+			xchan_query($items, true);
+			$items = fetch_post_tags($items);
+
+			$observer = App::get_observer();
+			$parent = $items[0];
+			$recips = (($parent['owner']['xchan_network'] === 'activitypub') ? get_iconfig($parent['id'], 'activitypub', 'recips', []) : []);
+			$to = (($recips && array_key_exists('to', $recips) && is_array($recips['to'])) ? $recips['to'] : null);
+			$nitems = [];
+			foreach ($items as $i) {
+				$mids = [];
+
+				if (intval($i['item_private'])) {
+					if (!$observer) {
+						continue;
+					}
+					/*
+					// ignore private reshare, possibly from hubzilla
+					if ($i['verb'] === 'Announce') {
+						if (!in_array($i['thr_parent'], $mids)) {
+							$mids[] = $i['thr_parent'];
+						}
+						continue;
+					}
+					// also ignore any children of the private reshares
+					if (in_array($i['thr_parent'], $mids)) {
+						continue;
+					}
+					*/
+
+					if ((!$to) || (!in_array($observer['xchan_url'], $to))) {
+						continue;
+					}
+				}
+				$nitems[] = $i;
+			}
+
+			if (!$nitems) {
+				http_status_exit(404, 'Not found');
+			}
+
+			App::set_pager_total($nitems[0]['total']);
+			App::set_pager_itemspage($itemspage);
+
+			$channel = channelx_by_n($nitems[0]['uid']);
+
+			if (!$channel) {
+				http_status_exit(404, 'Not found');
+			}
+
+			if (!perm_is_allowed($channel['channel_id'], get_observer_hash(), 'view_stream')) {
+				http_status_exit(403, 'Forbidden');
+			}
+
+			$i = Activity::encode_item_collection($nitems, App::$query_string, 'OrderedCollection', $nitems[0]['total']);
+
+			if (!$i) {
+				http_status_exit(404, 'Not found');
+			}
+
+			if ($portable_id && (!intval($nitems[0]['item_private']))) {
+				$c = q("select abook_id from abook where abook_channel = %d and abook_xchan = '%s'",
+					intval($nitems[0]['uid']),
+					dbesc($portable_id)
+				);
+				if (!$c) {
+					ThreadListener::store(z_root() . '/item/' . $item_id, $portable_id);
+				}
+			}
+
+			as_return_and_die($i, $channel);
+		}
+
+		goaway(z_root() . '/item/' . argv(1));
+	}
+}

Zotlabs/Module/Dircensor.php

@@ -4,6 +4,7 @@ namespace Zotlabs\Module;
 
 use App;
 use Zotlabs\Web\Controller;
+use Zotlabs\Lib\Libzotdir;
 
 
 class Dircensor extends Controller {
@@ -15,11 +16,12 @@ class Dircensor extends Controller {
 
 		$dirmode = intval(get_config('system','directory_mode'));
 
-		if (! ($dirmode == DIRECTORY_MODE_PRIMARY || $dirmode == DIRECTORY_MODE_STANDALONE)) {
+		if(!in_array($dirmode, [DIRECTORY_MODE_PRIMARY, DIRECTORY_MODE_SECONDARY, DIRECTORY_MODE_STANDALONE])) {
 			return;
 		}
 
 		$xchan = argv(1);
+
 		if(! $xchan) {
 			return;
 		}
@@ -32,19 +34,43 @@ class Dircensor extends Controller {
 			return;
 		}
 
-		$val = (($r[0]['xchan_censored']) ? 0 : 1);
+		$severity = intval($_REQUEST['severity'] ?? 0);
+		$flag = DIRECTORY_FLAG_OK;
 
-		q("update xchan set xchan_censored = $val where xchan_hash = '%s'",
+		if ($severity === 1) {
+			$flag = DIRECTORY_FLAG_UNSAFE;
+		}
+
+		if ($severity === 2) {
+			$flag = DIRECTORY_FLAG_HIDDEN;
+		}
+
+		Libzotdir::update($xchan, $r[0]['xchan_url'], true, $flag);
+
+		q("UPDATE xchan SET xchan_censored = %d WHERE xchan_hash = '%s'",
+			intval($flag),
 			dbesc($xchan)
 		);
 
-		if($val) {
+		q("UPDATE xtag SET xtag_flags = %d WHERE xtag_hash = '%s'",
+			intval($flag),
+			dbesc($xchan)
+		);
+
+		if($flag) {
 			info( t('Entry censored') . EOL);
 		}
 		else {
-			info( t('Entry uncensored') . EOL);
+			info( t('Entry OK') . EOL);
 		}
-		
+
+		if (isset($_REQUEST['aj'])) {
+			json_return_and_die([
+				'success' => 1,
+				'flag' => $flag
+			]);
+		}
+
 		goaway(z_root() . '/directory');
 
 	}

Zotlabs/Module/Directory.php

@@ -151,7 +151,7 @@ class Directory extends Controller {
 
 		$url = '';
 
-		if(($dirmode == DIRECTORY_MODE_PRIMARY) || ($dirmode == DIRECTORY_MODE_STANDALONE)) {
+		if(in_array($dirmode, [DIRECTORY_MODE_PRIMARY, DIRECTORY_MODE_SECONDARY, DIRECTORY_MODE_STANDALONE])) {
 			$url = z_root() . '/dirsearch';
 			if (is_site_admin()) {
 				$directory_admin = true;
@@ -191,7 +191,10 @@ class Directory extends Controller {
 			if(get_config('system','disable_directory_keywords'))
 				$kw = 0;
 
-			$query = $url . '?f=&kw=' . $kw . (($safe_mode != 1) ? '&safe=' . $safe_mode : '');
+			if (intval($safe_mode) === 0 && $directory_admin)
+				$safe_mode = -1;
+
+			$query = $url . '?f=&kw=' . $kw . (($safe_mode < 1) ? '&safe=' . $safe_mode : '');
 
 			if($token)
 				$query .= '&t=' . $token;
@@ -296,10 +299,11 @@ class Directory extends Controller {
 							$hometown = ((x($profile,'hometown') == 1) ? html2plain($profile['hometown'])  : False);
 
 							$about = ((x($profile,'about') == 1) ? zidify_links(bbcode($profile['about'], ['tryoembed' => false])) : False);
-							if ($about && $safe_mode) {
-								$about = html2plain($about);
+							if ($about && $safe_mode > 0) {
+								$about = strip_tags($about, '<br>');
 							}
 
+
 							$keywords = ((x($profile,'keywords')) ? $profile['keywords'] : '');
 
 
@@ -351,11 +355,15 @@ class Directory extends Controller {
 								'gender'   => $gender,
 								'pdesc'	=> $pdesc,
 								'pdesc_label' => t('Description:'),
-								'censor' => (($directory_admin) ? 'dircensor/' . $rr['hash'] : ''),
-								'censor_label' => (($rr['censored']) ? t('Uncensor') : t('Censor')),
+								'censor' => (($directory_admin) ? 'dircensor/' . $rr['hash'] . '?severity=' . ((intval($rr['censored']) > 0) ? 0 : 1) : ''),
+								'censor_label' => t('Unsafe'),
+								'censor_class' => ((intval($rr['censored']) === 1) ? 'active' : ''),
+								'censor_2' => (($directory_admin) ? 'dircensor/' . $rr['hash'] . '?severity=' . ((intval($rr['censored']) > 1) ? 0 : 2) : ''),
+								'censor_2_label' => t('Spam'),
+								'censor_2_class' => ((intval($rr['censored']) > 1) ? 'active' : ''),
 								'marital'  => $marital,
 								'homepage' => $homepage,
-								'homepageurl' => (($safe_mode) ? $homepageurl : linkify($homepageurl)),
+								'homepageurl' => (($safe_mode  > 0) ? $homepageurl : linkify($homepageurl)),
 								'hometown' => $hometown,
 								'hometown_label' => t('Hometown:'),
 								'about' => $about,
@@ -436,7 +444,8 @@ class Directory extends Controller {
 								'$reverse' => t('Reverse Alphabetic'),
 								'$date' => t('Newest to Oldest'),
 								'$reversedate' => t('Oldest to Newest'),
-								'$suggest' => $suggest ? '&suggest=1' : ''
+								'$suggest' => $suggest ? '&suggest=1' : '',
+								'$directory_admin' => $directory_admin
 							));
 
 

Zotlabs/Module/Dirsearch.php

@@ -15,8 +15,7 @@ class Dirsearch extends Controller {
 
 		$ret = array('success' => false);
 
-	//	logger('request: ' . print_r($_REQUEST,true));
-
+		// logger('request: ' . print_r($_REQUEST,true));
 
 		$dirmode = intval(get_config('system','directory_mode'));
 
@@ -25,7 +24,6 @@ class Dirsearch extends Controller {
 			json_return_and_die($ret);
 		}
 
-
 		$access_token = $_REQUEST['t'] ?? '';
 
 		$token = get_config('system','realm_token');
@@ -41,7 +39,8 @@ class Dirsearch extends Controller {
 		}
 
 		$sql_extra = '';
-
+		$keywords_query = '';
+		$hub_query = '';
 
 		$tables = array('name','address','locale','region','postcode','country','gender','marital','sexual','keywords');
 
@@ -85,9 +84,7 @@ class Dirsearch extends Controller {
 
 
 		// by default use a safe search
-		$safe     = ((x($_REQUEST,'safe')));    // ? intval($_REQUEST['safe'])  : 1 );
-		if ($safe === false)
-				$safe = 1;
+		$safe = $_REQUEST['safe'] ?? 1;
 
 		if(array_key_exists('sync',$_REQUEST)) {
 			if($_REQUEST['sync'])
@@ -102,10 +99,10 @@ class Dirsearch extends Controller {
 			$hub = \App::get_hostname();
 		}
 
-		if($hub)
-			$hub_query = " and xchan_hash in (select hubloc_hash from hubloc where hubloc_host =  '" . protect_sprintf(dbesc($hub)) . "') ";
-		else
-			$hub_query = '';
+		if($hub) {
+			$hub_query = " and xchan_hash in (select hubloc_hash from hubloc where hubloc_deleted = 0 and hubloc_host =  '" . protect_sprintf(dbesc($hub)) . "') ";
+		}
+
 
 		$sort_order  = ((x($_REQUEST,'order')) ? $_REQUEST['order'] : '');
 
@@ -131,9 +128,16 @@ class Dirsearch extends Controller {
 			$sql_extra .= $this->dir_query_build($joiner,'xprof_marital',$marital);
 		if($sexual)
 			$sql_extra .= $this->dir_query_build($joiner,'xprof_sexual',$sexual);
-		if($keywords)
+		if($keywords && $name) {
+			// this is a general search
 			$sql_extra .= $this->dir_query_build($joiner,'xprof_keywords',$keywords);
-
+		}
+		if($keywords && !$name) {
+			// this is a search for keywords only
+			$keywords_arr = explode(',', $keywords);
+			stringify_array_elms($keywords_arr, true);
+			$keywords_query = " AND xchan_hash IN (SELECT xtag_hash FROM xtag WHERE xtag_term IN (" . protect_sprintf(implode(',', $keywords_arr)) . ")) ";
+		}
 
 		// we only support an age range currently. You must set both agege
 		// (greater than or equal) and agele (less than or equal)
@@ -177,9 +181,15 @@ class Dirsearch extends Controller {
 			$sql_extra .= " and xchan_addr like '%%" . \App::get_hostname() . "' ";
 		}
 
-		$safesql = (($safe > 0) ? " and xchan_censored = 0 and xchan_selfcensored = 0 " : '');
+		$safesql = '';
+		if($safe > 0)
+			$safesql = " and xchan_censored = 0 and xchan_selfcensored = 0 ";
+
+		if($safe < 1)
+			$safesql = " and xchan_censored < 2 and xchan_selfcensored < 2 ";
+
 		if($safe < 0)
-			$safesql = " and ( xchan_censored = 1 OR xchan_selfcensored = 1 ) ";
+			$safesql = " and xchan_censored < 3 and xchan_selfcensored < 2 ";
 
 		if($forums)
 			$safesql .= " and xchan_pubforum = " . ((intval($forums)) ? '1 ' : '0 ');
@@ -215,26 +225,25 @@ class Dirsearch extends Controller {
 
 		if($sync) {
 			$spkt = array('transactions' => array());
-			$r = q("select * from updates where ud_date >= '%s' and ud_guid != '' and ud_addr != '' order by ud_date desc",
+
+			$r = q("SELECT * FROM updates WHERE ud_update = 0 AND ud_last = '%s' AND ud_date >= '%s' ORDER BY ud_date DESC",
+				dbesc(NULL_DATE),
 				dbesc($sync)
 			);
+
 			if($r) {
 				foreach($r as $rr) {
-					$flags = array();
-					if($rr['ud_flags'] & UPDATE_FLAGS_DELETED)
-						$flags[] = 'deleted';
-					if($rr['ud_flags'] & UPDATE_FLAGS_FORCED)
-						$flags[] = 'forced';
-
-					$spkt['transactions'][] = array(
+					$spkt['transactions'][] = [
 						'hash' => $rr['ud_hash'],
 						'address' => $rr['ud_addr'],
-						'transaction_id' => $rr['ud_guid'],
+						'host' => $rr['ud_host'],
+						'transaction_id' => $rr['ud_host'], // deprecated 2023-04-12 - can be removed after dir servers at version >= 8.4
 						'timestamp' => $rr['ud_date'],
-						'flags' => $flags
-					);
+						'flags' => $rr['ud_flags']
+					];
 				}
 			}
+
 			json_return_and_die($spkt);
 		}
 		else {
@@ -263,8 +272,8 @@ class Dirsearch extends Controller {
 				xprof.xprof_homepage as homepage,
 				xprof.xprof_hometown as hometown,
 				xprof.xprof_keywords as keywords
-				from xchan left join xprof on xchan_hash = xprof_hash left join hubloc on hubloc_hash = xchan_hash
-				where hubloc_primary = 1 and hubloc_updated > %s - INTERVAL %s and ( $logic $sql_extra ) $hub_query and xchan_network = 'zot6' and xchan_system = 0 and xchan_hidden = 0 and xchan_orphan = 0 and xchan_deleted = 0
+				from xchan left join xprof on xchan_hash = xprof_hash left join hubloc on (hubloc_id_url = xchan_url and hubloc_hash = xchan_hash)
+				where hubloc_primary = 1 and hubloc_updated > %s - INTERVAL %s and ( $logic $sql_extra ) $hub_query $keywords_query and xchan_network = 'zot6' and xchan_system = 0 and xchan_hidden = 0 and xchan_orphan = 0 and xchan_deleted = 0
 				$safesql $order $qlimit",
 				db_utcnow(),
 				db_quoteinterval('30 DAY')

Zotlabs/Module/Display.php

@@ -213,27 +213,27 @@ class Display extends \Zotlabs\Web\Controller {
 		$observer_hash = get_observer_hash();
 		$item_normal = item_normal();
 		$item_normal_update = item_normal_update();
-
-		$sql_extra = ((local_channel()) ? EMPTY_STR : item_permissions_sql(0, $observer_hash));
+		$sql_extra = '';
+		$r = [];
 
 		if($noscript_content || $load) {
-
 			require_once('include/channel.php');
 			$sys = get_sys_channel();
 			// in case somebody turned off public access to sys channel content using permissions
 			// make that content unsearchable by ensuring the owner uid can't match
 			$sys_id = perm_is_allowed($sys['channel_id'], $observer_hash, 'view_stream') ? $sys['channel_id'] : 0;
 
-			$r = null;
-
 			if(local_channel()) {
 				$r = q("SELECT item.id AS item_id FROM item WHERE uid = %d AND mid = '%s' $item_normal LIMIT 1",
 					intval(local_channel()),
 					dbesc($target_item['parent_mid'])
 				);
+
 			}
 
-			if($r === null) {
+			if(!$r) {
+				$sql_extra = item_permissions_sql(0, $observer_hash);
+
 				$r = q("SELECT item.id AS item_id FROM item
 					WHERE ((mid = '%s'
 					AND (((( item.allow_cid = '' AND item.allow_gid = '' AND item.deny_cid  = ''
@@ -257,7 +257,6 @@ class Display extends \Zotlabs\Web\Controller {
 			// make that content unsearchable by ensuring the owner uid can't match
 			$sys_id = perm_is_allowed($sys['channel_id'], $observer_hash, 'view_stream') ? $sys['channel_id'] : 0;
 
-			$r = null;
 			if(local_channel()) {
 				$r = q("SELECT item.parent AS item_id from item
 					WHERE uid = %d
@@ -270,7 +269,9 @@ class Display extends \Zotlabs\Web\Controller {
 				);
 			}
 
-			if($r === null) {
+			if(!$r) {
+				$sql_extra = item_permissions_sql(0, $observer_hash);
+
 				$r = q("SELECT item.id as item_id from item
 					WHERE ((parent_mid = '%s'
 					AND (((( item.allow_cid = '' AND item.allow_gid = '' AND item.deny_cid  = ''
@@ -287,10 +288,6 @@ class Display extends \Zotlabs\Web\Controller {
 			}
 		}
 
-		else {
-			$r = [];
-		}
-
 		if($r) {
 			$parents_str = ids_to_querystr($r,'item_id');
 			if($parents_str) {

Zotlabs/Module/Follow.php

@@ -70,12 +70,12 @@ class Follow extends Controller {
 		}
 
 		$uid = local_channel();
-		$url = notags(trim(punify($_REQUEST['url'])));
+		$url = notags(punify(trim($_REQUEST['url'])));
 		$return_url = $_SESSION['return_url'];
 		$interactive = $_REQUEST['interactive'] ?? 1;
 		$channel = App::get_channel();
 
-		$result = Connect::connect($channel,$url);
+		$result = Connect::connect($channel, $url);
 
 		if ($result['success'] == false) {
 			if ($result['message']) {

Zotlabs/Module/Hq.php

@@ -51,13 +51,20 @@ class Hq extends \Zotlabs\Web\Controller {
 		$o = '';
 
 		if($item_hash) {
+			// select the target item with a bias to our own item
+			$sql_order = ((local_channel() > $sys['channel_id']) ? 'DESC' : 'ASC');
 
-			$r = q("select id, uid, mid, parent_mid, thr_parent, verb, item_type, item_deleted, item_blocked from item where mid = '%s' limit 1",
+			$r = q("select id, uid, mid, parent_mid, thr_parent, verb, item_type, item_deleted, item_blocked from item where uid in (%d, %d) and mid = '%s' order by uid $sql_order limit 2",
+				intval(local_channel()),
+				intval($sys['channel_id']),
 				dbesc($item_hash)
 			);
 
 			if($r) {
 				$target_item = $r[0];
+				if (intval($target_item['uid']) === intval($sys['channel_id'])) {
+					$sys_item = true;
+				}
 			}
 
 			//if the item is to be moderated redirect to /moderate
@@ -158,19 +165,18 @@ class Hq extends \Zotlabs\Web\Controller {
 		}
 
 		if($load && $target_item) {
-			$r = null;
 
-			$r = q("SELECT item.id AS item_id FROM item
-				WHERE uid = %d
-				AND mid = '%s'
-				$item_normal
-				LIMIT 1",
-				intval(local_channel()),
-				dbesc($target_item['parent_mid'])
-			);
-
-			if(!$r) {
-				$sys_item = true;
+			if (!$sys_item) {
+				$r = q("SELECT item.id AS item_id FROM item
+					WHERE uid = %d
+					AND mid = '%s'
+					$item_normal
+					LIMIT 1",
+					intval(local_channel()),
+					dbesc($target_item['parent_mid'])
+				);
+			}
+			else {
 				$sql_extra = item_permissions_sql($sys['channel_id']);
 
 				$r = q("SELECT item.id AS item_id FROM item
@@ -184,20 +190,18 @@ class Hq extends \Zotlabs\Web\Controller {
 			}
 		}
 		elseif($update && $target_item) {
-			$r = null;
-
-			$r = q("SELECT item.parent AS item_id FROM item
-				WHERE uid = %d
-				AND parent_mid = '%s'
-				$item_normal_update
-				$simple_update
-				LIMIT 1",
-				intval(local_channel()),
-				dbesc($target_item['parent_mid'])
-			);
-
-			if(!$r) {
-				$sys_item = true;
+			if (!$sys_item) {
+				$r = q("SELECT item.parent AS item_id FROM item
+					WHERE uid = %d
+					AND parent_mid = '%s'
+					$item_normal_update
+					$simple_update
+					LIMIT 1",
+					intval(local_channel()),
+					dbesc($target_item['parent_mid'])
+				);
+			}
+			else {
 				$sql_extra = item_permissions_sql($sys['channel_id']);
 
 				$r = q("SELECT item.parent AS item_id FROM item
@@ -245,6 +249,7 @@ class Hq extends \Zotlabs\Web\Controller {
 
 		$options['offset'] = $_REQUEST['offset'] ?? 0;
 		$options['type'] = $_REQUEST['type'] ?? '';
+		$options['author'] = ((isset($_REQUEST['author'])) ? urldecode($_REQUEST['author']) : '');
 
 		$ret = Messages::get_messages_page($options);
 

Zotlabs/Module/Import.php

@@ -536,7 +536,7 @@ class Import extends Controller {
 				$since = datetime_convert(date_default_timezone_get(), date_default_timezone_get(), '0001-01-01 00:00');
 				$until = datetime_convert(date_default_timezone_get(), date_default_timezone_get(), 'now + 1 day');
 
-				$poll_interval = get_config('system', 'poll_interval', 3);
+				//$poll_interval = get_config('system', 'poll_interval', 3);
 				$page          = 0;
 
 				Master::Summon(['Content_importer', sprintf('%d', $page), $since, $until, $channel['channel_address'], urlencode($hz_server)]);

Zotlabs/Module/Invite.php

@@ -43,6 +43,19 @@ class Invite extends Controller {
 	const MYP = 	'ZAI';
 	const VERSION =	'2.0.0';
 
+	function init() {
+
+		if (!local_channel()) {
+			return;
+		}
+
+		$channel = App::get_channel();
+		if ($channel) {
+			profile_load($channel['channel_address']);
+		}
+
+	}
+
 	function post() {
 
 		// zai02

Zotlabs/Module/Item.php

@@ -63,8 +63,8 @@ class Item extends Controller {
 
 			// do we have the item (at all)?
 
-			$r = q("select * from item where mid = '%s' $item_normal limit 1",
-				dbesc(z_root() . '/item/' . $item_id)
+			$r = q("select parent_mid from item where uuid = '%s' $item_normal limit 1",
+				dbesc($item_id)
 			);
 
 			if (!$r) {
@@ -84,7 +84,7 @@ class Item extends Controller {
 				}
 				observer_auth($portable_id);
 
-				$i = q("select id as item_id from item where mid = '%s' $item_normal and owner_xchan = '%s' limit 1",
+				$i = q("select id as item_id, uid from item where mid = '%s' $item_normal and owner_xchan = '%s' limit 1",
 					dbesc($r[0]['parent_mid']),
 					dbesc($portable_id)
 				);
@@ -99,7 +99,7 @@ class Item extends Controller {
 			$sql_extra = item_permissions_sql(0);
 
 			if (!$i) {
-				$i = q("select id as item_id from item where mid = '%s' $item_normal $sql_extra order by item_wall desc limit 1",
+				$i = q("select id as item_id, uid from item where mid = '%s' $item_normal $sql_extra order by item_wall desc limit 1",
 					dbesc($r[0]['parent_mid'])
 				);
 			}
@@ -108,57 +108,53 @@ class Item extends Controller {
 				http_status_exit(403, 'Forbidden');
 			}
 
+			$chan = channelx_by_n($i[0]['uid']);
+
+			if (!$chan) {
+				http_status_exit(404, 'Not found');
+			}
+
+			if (!perm_is_allowed($chan['channel_id'], get_observer_hash(), 'view_stream')) {
+				http_status_exit(403, 'Forbidden');
+			}
+
 			$parents_str = ids_to_querystr($i, 'item_id');
 
-			$items = q("SELECT item.*, item.id AS item_id FROM item WHERE item.parent IN ( %s ) $item_normal order by item.id asc",
-				dbesc($parents_str)
+			$total = q("SELECT count(*) AS count FROM item WHERE parent = %d $item_normal",
+				intval($parents_str)
 			);
 
-			if (!$items) {
-				http_status_exit(404, 'Not found');
+			App::set_pager_total($total[0]['count']);
+			App::set_pager_itemspage(30);
+
+			if (App::$pager['total'] > App::$pager['itemspage']) {
+				// let mod conversation handle this request
+				App::$query_string = str_replace('item', 'conversation', App::$query_string);
+				$i = Activity::paged_collection_init(App::$pager['total'], App::$query_string);
+				as_return_and_die($i ,$chan);
 			}
+			else {
+				$items = q("SELECT item.*, item.id AS item_id FROM item WHERE item.parent = %d $item_normal ORDER BY item.id",
+					intval($parents_str)
+				);
 
-			xchan_query($items, true);
-			$items = fetch_post_tags($items, true);
+				xchan_query($items, true);
+				$items = fetch_post_tags($items, true);
 
-			if (!$items)
-				http_status_exit(404, 'Not found');
-
-			$chan = channelx_by_n($items[0]['uid']);
-
-			if (!$chan)
-				http_status_exit(404, 'Not found');
-
-			if (!perm_is_allowed($chan['channel_id'], get_observer_hash(), 'view_stream'))
-				http_status_exit(403, 'Forbidden');
-
-
-			$i = Activity::encode_item_collection($items, 'conversation/' . $item_id, 'OrderedCollection');
-
-			if (!$i)
-				http_status_exit(404, 'Not found');
+				$i = Activity::encode_item_collection($items, App::$query_string, 'OrderedCollection', App::$pager['total']);
+			}
 
 			if ($portable_id && (!intval($items[0]['item_private']))) {
-				ThreadListener::store(z_root() . '/item/' . $item_id, $portable_id);
+				$c = q("select abook_id from abook where abook_channel = %d and abook_xchan = '%s'",
+					intval($items[0]['uid']),
+					dbesc($portable_id)
+				);
+				if (!$c) {
+					ThreadListener::store(z_root() . '/item/' . $item_id, $portable_id);
+				}
 			}
 
-			$x = array_merge(['@context' => [
-				ACTIVITYSTREAMS_JSONLD_REV,
-				'https://w3id.org/security/v1',
-				z_root() . ZOT_APSCHEMA_REV
-			]], $i);
-
-			$headers                     = [];
-			$headers['Content-Type']     = 'application/x-zot+json';
-			$x['signature']              = LDSignatures::sign($x, $chan);
-			$ret                         = json_encode($x, JSON_UNESCAPED_SLASHES);
-			$headers['Digest']           = HTTPSig::generate_digest_header($ret);
-			$headers['(request-target)'] = strtolower($_SERVER['REQUEST_METHOD']) . ' ' . $_SERVER['REQUEST_URI'];
-			$h                           = HTTPSig::create_sig($headers, $chan['channel_prvkey'], channel_url($chan));
-			HTTPSig::set_headers($h);
-			echo $ret;
-			killme();
-
+			as_return_and_die($i ,$chan);
 		}
 
 		if (ActivityStreams::is_as_request()) {
@@ -181,8 +177,7 @@ class Item extends Controller {
 			// do we have the item (at all)?
 			// add preferential bias to item owners (item_wall = 1)
 
-			$r = q("select * from item where mid = '%s' or uuid = '%s' $item_normal order by item_wall desc limit 1",
-				dbesc(z_root() . '/item/' . $item_id),
+			$r = q("select * from item where uuid = '%s' $item_normal order by item_wall desc limit 1",
 				dbesc($item_id)
 			);
 
@@ -255,23 +250,7 @@ class Item extends Controller {
 				}
 			}
 
-			$x = array_merge(['@context' => [
-				ACTIVITYSTREAMS_JSONLD_REV,
-				'https://w3id.org/security/v1',
-				z_root() . ZOT_APSCHEMA_REV
-			]], $i);
-
-			$headers                     = [];
-			$headers['Content-Type']     = 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"';
-			$x['signature']              = LDSignatures::sign($x, $chan);
-			$ret                         = json_encode($x, JSON_UNESCAPED_SLASHES);
-			$headers['Date']             = datetime_convert('UTC', 'UTC', 'now', 'D, d M Y H:i:s \\G\\M\\T');
-			$headers['Digest']           = HTTPSig::generate_digest_header($ret);
-			$headers['(request-target)'] = strtolower($_SERVER['REQUEST_METHOD']) . ' ' . $_SERVER['REQUEST_URI'];
-			$h                           = HTTPSig::create_sig($headers, $chan['channel_prvkey'], channel_url($chan));
-			HTTPSig::set_headers($h);
-			echo $ret;
-			killme();
+			as_return_and_die($i ,$chan);
 
 		}
 
@@ -818,7 +797,7 @@ class Item extends Controller {
 
 		$is_group = get_pconfig($profile_uid, 'system', 'group_actor');
 
-		if (($is_group) && ($walltowall) && (!$walltowall_comment)) {
+		if ($is_group && $walltowall && !$walltowall_comment && !$webpage) {
 			$groupww           = true;
 			$str_contact_allow = $owner_xchan['xchan_hash'];
 			$str_group_allow   = '';
@@ -1194,12 +1173,15 @@ class Item extends Controller {
 				(($remote_id) ? $remote_id : basename($datarray['mid'])), true);
 		}
 
-
 		if ($orig_post) {
 			$datarray['id'] = $post_id;
 
 			$x = item_store_update($datarray, $execflag);
 
+			if ($x['success']) {
+				$this->add_listeners($datarray);
+			}
+
 			// We only need edit activities for other federated protocols
 			// which do not support edits natively. While this does federate
 			// edits, it presents a number of issues locally - such as #757 and #758.
@@ -1238,6 +1220,10 @@ class Item extends Controller {
 
 		$post = item_store($datarray, $execflag);
 
+		if ($post['success']) {
+			$this->add_listeners($datarray);
+		}
+
 		$post_id = $post['item_id'];
 
 		$datarray = $post['item'];
@@ -1518,8 +1504,9 @@ class Item extends Controller {
 
 		if (preg_match_all('/\[answer\](.*?)\[\/answer\]/ism', $body, $matches, PREG_SET_ORDER)) {
 			foreach ($matches as $match) {
-				$ptr[] = ['name' => $match[1], 'type' => 'Note', 'replies' => ['type' => 'Collection', 'totalItems' => 0]];
-				$body  = str_replace('[answer]' . $match[1] . '[/answer]', EMPTY_STR, $body);
+				$answer = escape_tags(trim($match[1]));
+				$ptr[] = ['name' => $answer, 'type' => 'Note', 'replies' => ['type' => 'Collection', 'totalItems' => 0]];
+				$body  = str_replace('[answer]' . $answer . '[/answer]', EMPTY_STR, $body);
 			}
 		}
 
@@ -1573,8 +1560,10 @@ class Item extends Controller {
 		$obj['content'] = bbcode($question);
 
 		foreach ($answers as $answer) {
-			if (trim($answer))
-				$ptr[] = ['name' => escape_tags($answer), 'type' => 'Note', 'replies' => ['type' => 'Collection', 'totalItems' => 0]];
+			$answer = escape_tags(trim($answer));
+			if ($answer) {
+				$ptr[] = ['name' => $answer, 'type' => 'Note', 'replies' => ['type' => 'Collection', 'totalItems' => 0]];
+			}
 		}
 
 		if ($multiple) {
@@ -1600,4 +1589,33 @@ class Item extends Controller {
 	}
 
 
+	function add_listeners($item) {
+		// ONLY public items!
+		if ($item['item_thread_top'] && !$item['item_private'] && !empty($item['term'])) {
+			foreach($item['term'] as $t) {
+				if (empty($t['url']) || $t['ttype'] != TERM_MENTION || $t['otype'] != TERM_OBJ_POST) {
+					continue;
+				}
+
+				$listener = q("select hubloc_hash, hubloc_network from hubloc where hubloc_id_url = '%s' and hubloc_deleted = 0 order by hubloc_id desc",
+					dbesc($t['url'])
+				);
+
+				if ($listener) {
+					$listener = Libzot::zot_record_preferred($listener);
+
+					$c = q("select abook_id from abook where abook_channel = %d and abook_xchan = '%s'",
+						intval($profile_uid),
+						dbesc($listener['hubloc_hash'])
+					);
+
+					if (!$c) {
+						ThreadListener::store($item['mid'], $listener['hubloc_hash']);
+					}
+				}
+			}
+		}
+	}
+
+
 }

Zotlabs/Module/Like.php

@@ -407,12 +407,20 @@ class Like extends Controller {
 					if (activity_match($rr['verb'], $activity))
 						$multi_undo = false;
 
+					$d = q("select * from item where id = %d",
+						intval($rr['id'])
+					);
+					if ($d) {
+						xchan_query($d);
+						$sync_item = fetch_post_tags($d);
+						Libsync::build_sync_packet($profile_uid, ['item' => [encode_item($sync_item[0], true)]]);
+					}
+
 					// drop_item was not done interactively, so we need to invoke the notifier
 					// in order to push the changes to connections
 
 					Master::Summon(array('Notifier', 'drop', $rr['id']));
 
-
 				}
 
 				if ($interactive)
@@ -443,7 +451,7 @@ class Like extends Controller {
 		}
 		else {
 			$post_type = (($item['resource_type'] === 'photo') ? t('photo') : t('status'));
-			if ($item['obj_type'] === ACTIVITY_OBJ_EVENT)
+			if (in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT]))
 				$post_type = t('event');
 
 			$obj_type = (($item['resource_type'] === 'photo') ? ACTIVITY_OBJ_PHOTO : ACTIVITY_OBJ_NOTE);
@@ -499,7 +507,7 @@ class Like extends Controller {
 			$private = (($public) ? 0 : 1);
 		}
 		else {
-			$arr['parent']     = $item['id'];
+			$arr['parent']     = $item['parent'];
 			$arr['thr_parent'] = $item['mid'];
 			$ulink             = '[zrl=' . $item_author['xchan_url'] . '][bdi]' . $item_author['xchan_name'] . '[/bdi][/zrl]';
 			$alink             = '[zrl=' . $observer['xchan_url'] . '][bdi]' . $observer['xchan_name'] . '[/bdi][/zrl]';
@@ -516,7 +524,7 @@ class Like extends Controller {
 		$arr['uid']          = $owner_uid;
 		$arr['item_flags']   = $item['item_flags'];
 		$arr['item_wall']    = $item['item_wall'];
-		$arr['parent_mid']   = (($extended_like) ? $arr['mid'] : $item['mid']);
+		$arr['parent_mid']   = (($extended_like) ? $arr['mid'] : $item['parent_mid']);
 		$arr['owner_xchan']  = (($extended_like) ? $ch[0]['xchan_hash'] : $thread_owner['xchan_hash']);
 		$arr['author_xchan'] = $observer['xchan_hash'];
 		$arr['body']         = sprintf($bodyverb, $alink, $ulink, $plink);
@@ -546,6 +554,7 @@ class Like extends Controller {
 		$arr['deny_cid']     = $deny_cid;
 		$arr['deny_gid']     = $deny_gid;
 		$arr['item_private'] = $private;
+		$arr['created']      = datetime_convert();
 
 		call_hooks('post_local', $arr);
 
@@ -561,6 +570,15 @@ class Like extends Controller {
 
 		call_hooks('post_local_end', $arr);
 
+		$r = q("select * from item where id = %d",
+			intval($post_id)
+		);
+		if ($r) {
+			xchan_query($r);
+			$sync_item = fetch_post_tags($r);
+			Libsync::build_sync_packet($profile_uid, ['item' => [encode_item($sync_item[0], true)]]);
+		}
+
 		if ($extended_like) {
 			$r = q("insert into likes (channel_id,liker,likee,iid,i_mid,verb,target_type,target_id,target) values (%d,'%s','%s',%d,'%s','%s','%s','%s','%s')",
 				intval($ch[0]['channel_id']),

Zotlabs/Module/Magic.php

@@ -16,7 +16,7 @@ class Magic extends Controller {
 			'url'     => '',
 			'message' => ''
 		];
-	
+
 		logger('mod_magic: invoked', LOGGER_DEBUG);
 
 		logger('args: ' . print_r($_REQUEST,true),LOGGER_DATA);
@@ -43,12 +43,12 @@ class Magic extends Controller {
 		$basepath = $parsed['scheme'] . '://' . $parsed['host'] . (isset($parsed['port']) ? ':' . $parsed['port'] : '');
 		$owapath = SConfig::get($basepath,'system','openwebauth', $basepath . '/owa');
 
-		// This is ready-made for a plugin that provides a blacklist or "ask me" before blindly authenticating. 
+		// This is ready-made for a plugin that provides a blacklist or "ask me" before blindly authenticating.
 		// By default, we'll proceed without asking.
 
 		$arr = [
 			'channel_id'  => local_channel(),
-			'destination' => $dest, 
+			'destination' => $dest,
 			'proceed'     => true
 		];
 

Zotlabs/Module/Notes.php

@@ -39,7 +39,7 @@ class Notes extends Controller {
 			}
 			set_pconfig(local_channel(),'notes','text',$body);
 
-			$ret['html'] = bbcode($body);
+			$ret['html'] = bbcode($body, ['tryoembed' => false]);
 			$ret['success'] = true;
 
 		}

Zotlabs/Module/Oep.php

@@ -343,6 +343,7 @@ class Oep extends \Zotlabs\Web\Controller {
 
 		if(! ($chn && $res))
 			return;
+
 		$c = q("select * from channel where channel_address = '%s' limit 1",
 			dbesc($chn)
 		);

Zotlabs/Module/Owa.php

@@ -32,7 +32,8 @@ class Owa extends Controller {
 				$keyId = $sigblock['keyId'];
 				if ($keyId) {
 					$r = q("SELECT * FROM hubloc LEFT JOIN xchan ON hubloc_hash = xchan_hash
-						WHERE hubloc_id_url = '%s' AND hubloc_deleted = 0 AND xchan_pubkey != '' ORDER BY hubloc_id DESC",
+						WHERE ( hubloc_addr = '%s' OR hubloc_id_url = '%s' ) AND hubloc_deleted = 0 AND xchan_pubkey != '' ORDER BY hubloc_id DESC",
+						dbesc(str_replace('acct:', '', $keyId)),
 						dbesc($keyId)
 					);
 					if (! $r) {

Zotlabs/Module/Pdledit.php

@@ -78,6 +78,23 @@ class Pdledit extends Controller {
 				}
 			}
 
+			// addons
+			$o .= '<h2>Addons</h2>';
+
+			$addons = plugins_installed_list();
+
+			foreach ($addons as $addon) {
+
+				$path = 'addon/' . $addon . '/Mod_' . ucfirst($addon) . '.php';
+
+				if (!file_exists($path))
+					continue;
+
+				$o .= '<a href="pdledit/' . $addon . '" >' . $addon . '</a>' . ((in_array($addon, $edited)) ? ' ' . t('(modified)') . ' <a href="pdledit/' . $addon . '/reset" >' . t('Reset') . '</a>': '' ) . '<br />';
+
+			}
+
+
 			$o .= '</div>';
 
 			// list module pdl files
@@ -85,11 +102,25 @@ class Pdledit extends Controller {
 		}
 
 		$t = get_pconfig(local_channel(),'system',$module);
-		$s = file_get_contents(theme_include($module));
-		if(! $t) {
+		$s = '';
+
+		if(!$t) {
+			$sys_path = theme_include($module);
+
+			if ($sys_path) {
+				$s = file_get_contents($sys_path);
+			}
+			else {
+				$addon_path = 'addon/' . argv(1) . '/' . $module;
+				if (file_exists($addon_path)) {
+					$s = file_get_contents($addon_path);
+				}
+			}
+
 			$t = $s;
 		}
-		if(! $t) {
+
+		if(!$t) {
 			notice( t('Layout not found.') . EOL);
 			return '';
 		}

Zotlabs/Module/Pdledit_gui.php

@@ -6,6 +6,7 @@ use App;
 use Zotlabs\Web\Controller;
 use Zotlabs\Render\Comanche;
 use Zotlabs\Lib\Libsync;
+use Zotlabs\Lib\Apps;
 
 class Pdledit_gui extends Controller {
 
@@ -220,6 +221,7 @@ class Pdledit_gui extends Controller {
 
 	function get_modules() {
 		$ret = '';
+		$arr = [];
 
 		$files = glob('Zotlabs/Module/*.php');
 		if($files) {
@@ -232,23 +234,61 @@ class Pdledit_gui extends Controller {
 
 				$x = theme_include('mod_' . $name . '.pdl');
 				if($x) {
-					$ret .= '<div class="mb-2"><a href="pdledit_gui/' . $name . '">' . $name . '</a></div>';
+					$arr[] = $name;
 				}
 			}
 		}
 
+		$addons = plugins_installed_list();
+		if ($addons) {
+			foreach ($addons as $name) {
+				if (!Apps::addon_app_installed(local_channel(), $name)) {
+					continue;
+				}
+
+				$path = 'addon/' . $name . '/mod_' . $name . '.pdl';
+				if (file_exists($path)) {
+					$arr[] = $name;
+				}
+			}
+		}
+
+		sort($arr);
+
+		foreach ($arr as $name) {
+			$ret .= '<div class="mb-2"><a href="pdledit_gui/' . $name . '">' . $name . '</a></div>';
+		}
+
 		return $ret;
+
 	}
 
 	function get_widgets($module) {
 		$ret = [];
 
+
 		$checkpaths = [
 			'Zotlabs/Widget/*.php'
 		];
 
+		$addons = plugins_installed_list();
+
+		if ($addons) {
+			foreach ($addons as $name) {
+				if (!Apps::addon_app_installed(local_channel(), $name)) {
+					continue;
+				}
+
+				$path = 'addon/' . $name . '/Widget';
+				if (is_dir($path)) {
+					$checkpaths[] = $path . '/*.php';
+				}
+			}
+		}
+
 		foreach ($checkpaths as $path) {
 			$files = glob($path);
+
 			if($files) {
 				foreach($files as $f) {
 					$name = lcfirst(basename($f, '.php'));
@@ -272,6 +312,8 @@ class Pdledit_gui extends Controller {
 			}
 		}
 
+		usort($ret, fn($a, $b) => $a['name'] <=> $b['name']);
+
 		return $ret;
 	}
 
@@ -536,12 +578,21 @@ class Pdledit_gui extends Controller {
 			'modified' => true
 		];
 
-		$pdl_path = 'mod_' . $module . '.pdl';
+		$pdl = 'mod_' . $module . '.pdl';
+		$pdl_path = '';
 
-		$ret['pdl'] = get_pconfig(local_channel(), 'system', $pdl_path);
+		$ret['pdl'] = get_pconfig(local_channel(), 'system', $pdl);
 
 		if(!$ret['pdl']) {
-			$pdl_path = theme_include($pdl_path);
+			$pdl_path = theme_include($pdl);
+
+			if (!$pdl_path) {
+				$addon_path = 'addon/' . $module . '/' . $pdl;
+				if (file_exists($addon_path)) {
+					$pdl_path = $addon_path;
+				}
+			}
+
 			if ($pdl_path) {
 				$ret['pdl'] = file_get_contents($pdl_path);
 				$ret['modified'] = false;

Zotlabs/Module/Photo.php

@@ -271,7 +271,6 @@ class Photo extends \Zotlabs\Web\Controller {
  		    $modified = time();
 
 		header_remove('Pragma');
-
 		if((isset($_SERVER['HTTP_IF_NONE_MATCH']) && $_SERVER['HTTP_IF_NONE_MATCH'] === $etag) || (!isset($_SERVER['HTTP_IF_NONE_MATCH']) && isset($_SERVER['HTTP_IF_MODIFIED_SINCE']) && $_SERVER['HTTP_IF_MODIFIED_SINCE'] === gmdate("D, d M Y H:i:s", $modified) . " GMT")) {
 			header_remove('Expires');
 			header_remove('Cache-Control');
@@ -307,6 +306,8 @@ class Photo extends \Zotlabs\Web\Controller {
 			// This has performance considerations but we highly recommend you
 			// leave it alone.
 
+
+
 			$maxage = $cache_mode['age'];
 
 			if($cache_mode['exp'] || (! isset($expires)) || (isset($expires) && $expires - 60 < time()))

Zotlabs/Module/Profiles.php

@@ -217,7 +217,6 @@ class Profiles extends \Zotlabs\Web\Controller {
 
 			check_form_security_token_redirectOnErr('/profiles', 'profile_edit');
 
-
 			$is_default = (($orig[0]['is_default']) ? 1 : 0);
 
 			$profile_name = notags(trim($_POST['profile_name']));
@@ -579,12 +578,12 @@ class Profiles extends \Zotlabs\Web\Controller {
 			$channel = \App::get_channel();
 
 			if($namechanged && $is_default) {
-				// change name on all associated xchans by matching the url
-				q("UPDATE xchan SET xchan_name = '%s', xchan_name_date = '%s' WHERE xchan_url = '%s'",
+				q("UPDATE xchan SET xchan_name = '%s', xchan_name_date = '%s' WHERE xchan_hash = '%s'",
 					dbesc($name),
 					dbesc(datetime_convert()),
-					dbesc(z_root() . '/channel/' . $channel['channel_address'])
+					dbesc($channel['xchan_hash'])
 				);
+
 				q("UPDATE channel SET channel_name = '%s' WHERE channel_hash = '%s'",
 					dbesc($name),
 					dbesc($channel['xchan_hash'])

Zotlabs/Module/Pubstream.php

@@ -12,6 +12,7 @@ class Pubstream extends \Zotlabs\Web\Controller {
 
 	function get($update = 0, $load = false) {
 
+
 		if(local_channel()) {
 			if(! Apps::system_app_installed(local_channel(), 'Public Stream')) {
 				//Do not display any associated widgets at this point
@@ -31,18 +32,16 @@ class Pubstream extends \Zotlabs\Web\Controller {
 			}
 		}
 
-		$site_firehose = ((intval(get_config('system','site_firehose',0))) ? true : false);
 		$net_firehose  = ((get_config('system','disable_discover_tab',1)) ? false : true);
 
-		if(! ($site_firehose || $net_firehose)) {
+		if(!$net_firehose) {
 			return '';
 		}
 
-		if($net_firehose) {
-			$site_firehose = false;
-		}
+		$site_firehose = ((intval(get_config('system','site_firehose',0))) ? true : false);
+
+		$mid = (($_REQUEST['mid']) ? unpack_link_id($_REQUEST['mid']) : '');
 
-		$mid = ((x($_REQUEST, 'mid')) ? unpack_link_id($_REQUEST['mid']) : '');
 		if ($mid === false) {
 			notice(t('Malformed message id.') . EOL);
 			return;
@@ -160,15 +159,15 @@ class Pubstream extends \Zotlabs\Web\Controller {
 		require_once('include/security.php');
 
 		$sys = get_sys_channel();
+		$uids = " and item.uid  = " . intval($sys['channel_id']) . " ";
 		$abook_uids = " and abook.abook_channel = " . intval($sys['channel_id']) . " ";
+		$sql_extra = item_permissions_sql($sys['channel_id']);
+		$sql_extra_order = '';
+		$site_firehose_sql = '';
+		$thread_top =  " and item.item_thread_top = 1 ";
 
 		if($site_firehose) {
-			$uids = " and item.uid in ( " . stream_perms_api_uids(PERMS_PUBLIC) . " ) and item_private = 0  and item_wall = 1 ";
-		}
-		else {
-			$uids = " and item.uid  = " . intval($sys['channel_id']) . " ";
-			$sql_extra = item_permissions_sql($sys['channel_id']);
-			\App::$data['firehose'] = intval($sys['channel_id']);
+			$site_firehose_sql = " and owner_xchan in (select channel_hash from channel where channel_system = 0 and channel_removed = 0) ";
 		}
 
 		if(get_config('system','public_list_mode'))
@@ -179,6 +178,8 @@ class Pubstream extends \Zotlabs\Web\Controller {
 
 		if(x($hashtags)) {
 			$sql_extra .= protect_sprintf(term_query('item', $hashtags, TERM_HASHTAG, TERM_COMMUNITYTAG));
+			$sql_extra_order = " ORDER BY item.created DESC ";
+			$thread_top = '';
 		}
 
 		$net_query = (($net) ? " left join xchan on xchan_hash = author_xchan " : '');
@@ -200,7 +201,9 @@ class Pubstream extends \Zotlabs\Web\Controller {
 					$r = q("SELECT parent AS item_id FROM item
 						left join abook on item.author_xchan = abook.abook_xchan
 						$net_query
-						WHERE mid = '%s' $uids $item_normal
+						WHERE item.mid = '%s' and item.item_private = 0
+						$uids $site_firehose_sql
+						$item_normal
 						and (abook.abook_blocked = 0 or abook.abook_flags is null)
 						$sql_extra $net_query2",
 						dbesc($mid)
@@ -208,10 +211,12 @@ class Pubstream extends \Zotlabs\Web\Controller {
 				}
 				else {
 					// Fetch a page full of parent items for this page
-					$r = dbq("SELECT item.id AS item_id FROM item
+					$r = dbq("SELECT parent AS item_id FROM item
 						left join abook on ( item.author_xchan = abook.abook_xchan $abook_uids )
 						$net_query
-						WHERE true $uids and item.item_thread_top = 1 $item_normal
+						WHERE item.item_private = 0 $thread_top
+						$uids $site_firehose_sql
+						$item_normal
 						and (abook.abook_blocked = 0 or abook.abook_flags is null)
 						$sql_extra $net_query2
 						ORDER BY $ordering DESC $pager_sql "
@@ -223,7 +228,8 @@ class Pubstream extends \Zotlabs\Web\Controller {
 					$r = q("SELECT parent AS item_id FROM item
 						left join abook on item.author_xchan = abook.abook_xchan
 						$net_query
-						WHERE mid = '%s' $uids $item_normal_update $simple_update
+						WHERE item.mid = '%s' and item.item_private = 0
+						$uids $site_firehose_sql $item_normal_update $simple_update
 						and (abook.abook_blocked = 0 or abook.abook_flags is null)
 						$sql_extra $net_query2",
 						dbesc($mid)
@@ -233,7 +239,8 @@ class Pubstream extends \Zotlabs\Web\Controller {
 					$r = dbq("SELECT parent AS item_id FROM item
 						left join abook on item.author_xchan = abook.abook_xchan
 						$net_query
-						WHERE true $uids $item_normal_update
+						WHERE item.item_private = 0 $thread_top
+						$uids $site_firehose_sql $item_normal_update
 						$simple_update
 						and (abook.abook_blocked = 0 or abook.abook_flags is null)
 						$sql_extra $net_query2"
@@ -251,21 +258,26 @@ class Pubstream extends \Zotlabs\Web\Controller {
 				$items = dbq("SELECT item.*, item.id AS item_id FROM item
 					WHERE true $uids $item_normal
 					AND item.parent IN ( $parents_str )
-					$sql_extra"
+					$sql_extra $sql_extra_order"
 				);
 
+
 				// use effective_uid param of xchan_query to help sort out comment permission
 				// for sys_channel owned items.
 
-				xchan_query($items,true,(($sys) ? local_channel() : 0));
+				xchan_query($items, true, local_channel());
 				$items = fetch_post_tags($items,true);
-				$items = conv_sort($items,$ordering);
+
+				if (!$hashtags) {
+					$items = conv_sort($items, $ordering);
+				}
+
+
 			}
 
 		}
 
-		// fake it
-		$mode = (($hashtags) ? 'search' : 'pubstream');
+		$mode = (($hashtags) ? 'pubstream-new' : 'pubstream');
 
 		$o .= conversation($items,$mode,$update,$page_mode);
 

Zotlabs/Module/Search.php

@@ -58,7 +58,7 @@ class Search extends Controller {
 
 		$o .= search($search, 'search-box', '/search', ((local_channel()) ? true : false));
 
-		if (local_channel() && strpos($search, 'https://') === 0 && !$update && !$load) {
+		if (local_channel() && str_starts_with($search, 'https://') && !$update && !$load) {
 
 			$url = htmlspecialchars_decode($search);
 
@@ -75,7 +75,7 @@ class Search extends Controller {
 			if ($f) {
 				$mid = $f[0]['message_id'];
 				foreach ($f as $m) {
-					if (strpos($search, $m['message_id']) === 0) {
+					if (str_starts_with($url, $m['message_id'])) {
 						$mid = $m['message_id'];
 						break;
 					}

Zotlabs/Module/Settings/Account.php

@@ -6,11 +6,11 @@ class Account {
 
 	function post() {
 		check_form_security_token_redirectOnErr('/settings/account', 'settings_account');
-		
+
 		call_hooks('account_settings_post', $_POST);
-	
+
 		$errs = array();
-	
+
 		$email = ((x($_POST,'email')) ? trim(notags($_POST['email'])) : '');
 
 		$account = \App::get_account();
@@ -34,38 +34,38 @@ class Account {
 				}
 			}
 		}
-	
+
 		if($errs) {
 			foreach($errs as $err)
 				notice($err . EOL);
 			$errs = array();
 		}
-	
-	
+
+
 		if((x($_POST,'npassword')) || (x($_POST,'confirm'))) {
-	
+
 			$origpass = trim($_POST['origpass']);
-	
+
 			require_once('include/auth.php');
 			if(! account_verify_password($email,$origpass)) {
 				$errs[] = t('Password verification failed.');
 			}
-	
+
 			$newpass = trim($_POST['npassword']);
 			$confirm = trim($_POST['confirm']);
-	
+
 			if($newpass != $confirm ) {
 				$errs[] = t('Passwords do not match. Password unchanged.');
 			}
-	
+
 			if((! x($newpass)) || (! x($confirm))) {
 				$errs[] = t('Empty passwords are not allowed. Password unchanged.');
 			}
-	
+
 			if(! $errs) {
 				$salt = random_string(32);
 				$password_encoded = hash('whirlpool', $salt . $newpass);
-				$r = q("update account set account_salt = '%s', account_password = '%s', account_password_changed = '%s' 
+				$r = q("update account set account_salt = '%s', account_password = '%s', account_password_changed = '%s'
 					where account_id = %d",
 					dbesc($salt),
 					dbesc($password_encoded),
@@ -78,36 +78,38 @@ class Account {
 					$errs[] = t('Password update failed. Please try again.');
 			}
 		}
-	
-	
+
+
 		if($errs) {
 			foreach($errs as $err)
 				notice($err . EOL);
 		}
 		goaway(z_root() . '/settings/account' );
 	}
-	
 
-	
+
+
 	function get() {
 		$account_settings = "";
-			
+
 		call_hooks('account_settings', $account_settings);
-	
+
 		$email      = \App::$account['account_email'];
-		$attremail  = (!strpos($email, '@')) ? 'disabled="disabled"' : '';
+		$attremail  = ((!strpos($email, '@')) ? 'disabled="disabled"' : '');
 
 		$tpl = get_markup_template("settings_account.tpl");
 		$o .= replace_macros($tpl, array(
 			'$form_security_token' => get_form_security_token("settings_account"),
-			'$title'	=> t('Account Settings'),
-			'$origpass' => array('origpass', t('Current Password'), ' ',''),
-			'$password1'=> array('npassword', t('Enter New Password'), '', ''),
-			'$password2'=> array('confirm', t('Confirm New Password'), '', t('Leave password fields blank unless changing')),
-			'$submit' 	=> t('Submit'),
-			'$email' 	=> array('email', t('DId2 or Email Address:'), $email, '', '', $attremail),
-			'$removeme' => t('Remove Account'),
-			'$removeaccount' => t('Remove this account including all its channels'),
+			'$title'            => t('Account Settings'),
+			'$origpass'         => array('origpass', t('Current Password'), ' ',''),
+			'$password1'        => array('npassword', t('Enter New Password'), '', ''),
+			'$password2'        => array('confirm', t('Confirm New Password'), '', t('Leave password fields blank unless changing')),
+			'$submit'           => t('Submit'),
+			'$mfa'              => t('Multi-Factor Authentication'),
+			'$email'            => array('email', t('DId2 or Email Address:'), $email, '', '', $attremail),
+			'$email_hidden'     => (($attremail) ? $email : ''),
+			'$removeme'         => t('Remove Account'),
+			'$removeaccount'    => t('Remove this account including all its channels'),
 			'$account_settings' => $account_settings
 		));
 		return $o;

Zotlabs/Module/Settings/Multifactor.php

@@ -0,0 +1,90 @@
+<?php
+
+namespace Zotlabs\Module\Settings;
+
+use App;
+use chillerlan\QRCode\QRCode;
+use Zotlabs\Lib\AConfig;
+use Zotlabs\Lib\System;
+use OTPHP\TOTP;
+use ParagonIE\ConstantTime\Base32;
+
+
+class Multifactor {
+	public function post() {
+		check_form_security_token_redirectOnErr('/settings/multifactor', 'settings_mfa');
+
+		$account = App::get_account();
+		if (!$account) {
+			return;
+		}
+
+		if (empty($_POST['password'])) {
+			notice(t('Password is required') . EOL);
+			return;
+		}
+
+		$password = trim($_POST['password']);
+		if(!account_verify_password($account['account_email'], $password)) {
+			notice(t('The provided password is not correct') . EOL);
+			return;
+		}
+
+		$enable_mfa = isset($_POST['enable_mfa']) ? (int) $_POST['enable_mfa'] : false;
+		AConfig::Set($account['account_id'], 'system', 'mfa_enabled', $enable_mfa);
+		if ($enable_mfa) {
+			$_SESSION['2FA_VERIFIED'] = true;
+		}
+	}
+
+	public function get() {
+		$account = App::get_account();
+		if (!$account) {
+			return '';
+		}
+
+		if (!$account['account_external']) {
+			$otp = TOTP::create();
+			$otp->setLabel($account['account_email']);
+			// $otp->setLabel(rawurlencode(System::get_platform_name()));
+			$otp->setIssuer(rawurlencode(System::get_platform_name()));
+
+			$mySecret = trim(Base32::encodeUpper(random_bytes(32)), '=');
+			$otp = TOTP::create($mySecret);
+			q("UPDATE account set account_external = '%s' where account_id = %d",
+				dbesc($otp->getSecret()),
+				intval($account['account_id'])
+			);
+			$account['account_external'] = $otp->getSecret();
+		}
+
+		$otp = TOTP::create($account['account_external']);
+		$otp->setLabel($account['account_email']);
+		$otp->setIssuer(rawurlencode(System::get_platform_name()));
+		$uri = $otp->getProvisioningUri();
+		return replace_macros(get_markup_template('totp_setup.tpl'),
+			[
+				'$form_security_token' => get_form_security_token("settings_mfa"),
+				'$title' => t('Account Multi-Factor Authentication'),
+				'$secret_text' => t('This is your generated secret. It may be used in some cases if the QR image cannot be read. Please store it in a safe place.'),
+				'$test_title' => t('Please enter the code from your authenticator app'),
+				'$test_title_sub' => t('You will only be able to enable MFA if the test passes'),
+				'$qrcode' => (new QRCode())->render($uri),
+				'$uri' => $uri,
+				'$secret' => ($account['account_external'] ?? ''),
+				'$test_pass' => t("Congratulations, the provided code was correct"),
+				'$test_fail' => t("Incorrect code"),
+				'$enable_mfa' => [
+					'enable_mfa',
+					t('Enable Multi-Factor Authentication'),
+					AConfig::Get($account['account_id'], 'system', 'mfa_enabled'),
+					t('Logging in will require you to be in possession of your smartphone'),
+					[t('No'), t('Yes')]
+				],
+				'$password' => ['password', t('Your account password'), '', t('Required')],
+				'$submit' => t('Submit'),
+				'$test' => t('Test')
+			]
+		);
+	}
+}

Zotlabs/Module/Settings/Privacy.php

@@ -22,6 +22,9 @@ class Privacy {
 		$autoperms = (((x($_POST, 'autoperms')) && (intval($_POST['autoperms']) == 1)) ? 1 : 0);
 		set_pconfig(local_channel(), 'system', 'autoperms', $autoperms);
 
+		$permit_all_mentions = (((x($_POST, 'permit_all_mentions')) && (intval($_POST['permit_all_mentions']) == 1)) ? 1 : 0);
+		set_pconfig(local_channel(), 'system', 'permit_all_mentions', $permit_all_mentions);
+
 		$role = get_pconfig(local_channel(), 'system', 'permissions_role');
 		if ($role === 'custom') {
 
@@ -99,6 +102,7 @@ class Privacy {
 		$autoperms = get_pconfig(local_channel(), 'system', 'autoperms');
 		$index_opt_out = get_pconfig(local_channel(), 'system', 'index_opt_out');
 		$group_actor = get_pconfig(local_channel(), 'system', 'group_actor');
+		$permit_all_mentions = get_pconfig(local_channel(), 'system', 'permit_all_mentions');
 
 		$permissions_role   = get_pconfig(local_channel(), 'system', 'permissions_role', 'custom');
 		$permission_limits  = ($permissions_role === 'custom');
@@ -119,7 +123,8 @@ class Privacy {
 			],
 			'$autoperms' => ['autoperms', t('Automatically approve new contacts'), $autoperms, '', [t('No'), t('Yes')]],
 			'$index_opt_out' => ['index_opt_out', t('Opt-out of search engine indexing'), $index_opt_out, '', [t('No'), t('Yes')]],
-			'$group_actor' => ['group_actor', t('Group actor'), $group_actor, t('Allow this channel to act as a forum'), [t('No'), t('Yes')]]
+			'$group_actor' => ['group_actor', t('Group actor'), $group_actor, t('Allow this channel to act as a forum'), [t('No'), t('Yes')]],
+			'$permit_all_mentions' => ['permit_all_mentions', t('Accept all messages which mention you'), $permit_all_mentions, t('This setting bypasses normal permissions'), [t('No'), t('Yes')]],
 		]);
 
 		return $o;

Zotlabs/Module/Sharedwithme.php

@@ -19,13 +19,13 @@ class Sharedwithme extends Controller {
 			notice( t('Permission denied.') . EOL);
 			return;
 		}
-		
+
 		$channel = \App::get_channel();
-	
+
 		$is_owner = (local_channel() && (local_channel() == $channel['channel_id']));
 
 		$item_normal = item_normal();
-	
+
 		//drop single file - localuser
 		if((argc() > 2) && (argv(2) === 'drop')) {
 
@@ -36,7 +36,7 @@ class Sharedwithme extends Controller {
 			goaway(z_root() . '/sharedwithme');
 
 		}
-	
+
 		//drop all files - localuser
 		if((argc() > 1) && (argv(1) === 'dropall')) {
 
@@ -62,33 +62,34 @@ class Sharedwithme extends Controller {
 			dbesc($channel['channel_hash'])
 		);
 
+		$r = fetch_post_tags($r, true);
+
 		$items = [];
 		$ids = [];
 
 		if($r) {
-	
+
 			foreach($r as $rr) {
-				$object = json_decode($rr['obj'],true);
-				$meta = self::get_meta($object);
+				$meta = get_iconfig($rr, 'attach', 'meta');
 
 				$item = [];
 				$item['id'] = $rr['id'];
 				$item['objfiletype'] = $meta['type'];
 				$item['objfiletypeclass'] = getIconFromType($meta['type']);
 				$item['objurl'] = $meta['path'] . '?f=&zid=' . $channel['xchan_addr'];
-				$item['objfilename'] = $object['name'];
+				$item['objfilename'] = $meta['name'];
 				$item['objfilesize'] = userReadableSize($meta['size']);
 				$item['objedited'] = $meta['edited'];
 				$item['unseen'] = $rr['item_unseen'];
-	
+
 				$items[] = $item;
-	
+
 				if($item['unseen']) {
 					$ids[] = $rr['id'];
 				}
-	
+
 			}
-	
+
 		}
 
 		$ids = implode(',', $ids);
@@ -98,9 +99,9 @@ class Sharedwithme extends Controller {
 				intval(local_channel())
 			);
 		}
-	
+
 		$o = '';
-	
+
 		$o .= replace_macros(get_markup_template('sharedwithme.tpl'), array(
 			'$header' => t('Files: shared with me'),
 			'$name' => t('Name'),
@@ -111,27 +112,9 @@ class Sharedwithme extends Controller {
 			'$drop' => t('Remove this file'),
 			'$items' => $items
 		));
-	
+
 		return $o;
-	
-	}
-	
-	function get_meta($object) {
-
-		$ret = [];
-
-		if(! is_array($object['attachment']))
-			return;
-
-		foreach($object['attachment'] as $a) {
-			if($a['name'] === 'zot.attach.meta') {
-				$ret = $a['value'];
-				break;
-			}
-		}
-
-		return $ret;
 
 	}
-	
+
 }

Zotlabs/Module/Sse.php

@@ -50,7 +50,7 @@ class Sse extends Controller {
 			self::$vnotify = get_pconfig(self::$uid, 'system', 'vnotify');
 		}
 
-		$sleep_seconds = 3;
+		$sleep = 1000000; // microseconds
 
 		self::$sse_enabled = get_config('system', 'sse_enabled', 0);
 
@@ -63,17 +63,24 @@ class Sse extends Controller {
 			header("Connection: keep-alive");
 			header("X-Accel-Buffering: no");
 
+			$i = 0;
+
 			while(true) {
 
-				if(! self::$sse_id) {
-
-					// Update chat presence indication
+				// reset counter for updating chatpresence about every minute
+				if (($i * $sleep)/60 > 1000000) {
+					$i = 0;
+				}
 
+				if(!self::$sse_id && $i === 0) {
+					// Update chat presence indication about once per minute
 					$r = q("select cp_id, cp_room from chatpresence where cp_xchan = '%s' and cp_client = '%s' and cp_room = 0 limit 1",
 						dbesc(self::$ob_hash),
 						dbesc($_SERVER['REMOTE_ADDR'])
 					);
+
 					$basic_presence = false;
+
 					if($r) {
 						$basic_presence = true;
 						q("update chatpresence set cp_last = '%s' where cp_id = %d",
@@ -81,7 +88,8 @@ class Sse extends Controller {
 							intval($r[0]['cp_id'])
 						);
 					}
-					if(! $basic_presence) {
+
+					if(!$basic_presence) {
 						q("insert into chatpresence ( cp_xchan, cp_last, cp_status, cp_client)
 							values( '%s', '%s', '%s', '%s' ) ",
 							dbesc(self::$ob_hash),
@@ -94,18 +102,23 @@ class Sse extends Controller {
 
 				XConfig::Load(self::$ob_hash);
 
-				$result = XConfig::Get(self::$ob_hash, 'sse', 'notifications', []);
+				$result = [];
 				$lock = XConfig::Get(self::$ob_hash, 'sse', 'lock');
 
+				if (!$lock) {
+					$result = XConfig::Get(self::$ob_hash, 'sse', 'notifications', []);
+				}
+
+
 				// We do not have the local_channel in the addon.
 				// Reset pubs here if the app is not installed.
 				if (self::$uid && (!(self::$vnotify & VNOTIFY_PUBS) || !Apps::system_app_installed(self::$uid, 'Public Stream'))) {
-					$result['pubs']['count'] = 0;
-					$result['pubs']['notifications'] = [];
-					$result['pubs']['offset'] = -1;
+					if (isset($result['pubs'])) {
+						unset($result['pubs']);
+					}
 				}
 
-				if($result && !$lock) {
+				if($result) {
 					echo "event: notifications\n";
 					echo 'data: ' . json_encode($result);
 					echo "\n\n";
@@ -121,6 +134,7 @@ class Sse extends Controller {
 
 				if(ob_get_length() > 0)
 					ob_end_flush();
+
 				flush();
 
 				if(connection_status() != CONNECTION_NORMAL || connection_aborted()) {
@@ -129,7 +143,9 @@ class Sse extends Controller {
 					break;
 				}
 
-				sleep($sleep_seconds);
+				$i++;
+
+				usleep($sleep);
 
 			}
 

Zotlabs/Module/Sse_bs.php

@@ -173,6 +173,9 @@ class Sse_bs extends Controller {
 
 		$item_normal = item_normal();
 
+		// FEP-5624 filter approvals for comments
+		$approvals = " AND verb NOT IN ('" . dbesc(ACTIVITY_ATTEND) . "', 'Accept', '" . dbesc(ACTIVITY_ATTENDNO) . "', 'Reject') ";
+
 		if ($notifications) {
 			$items = q("SELECT * FROM item
 				WHERE uid = %d
@@ -181,6 +184,7 @@ class Sse_bs extends Controller {
 				AND obj_type NOT IN ('Document', 'Video', 'Audio', 'Image')
 				AND author_xchan != '%s'
 				$item_normal
+				$approvals
 				$sql_extra
 				$sql_extra2
 				ORDER BY created DESC LIMIT $limit OFFSET $offset",
@@ -205,18 +209,19 @@ class Sse_bs extends Controller {
 
 		}
 
-		$r = q("SELECT count(id) as total FROM item
+		$r = q("SELECT id FROM item
 			WHERE uid = %d and item_unseen = 1 AND item_wall = 0 AND item_private IN (0, 1)
 			AND obj_type NOT IN ('Document', 'Video', 'Audio', 'Image')
 			AND author_xchan != '%s'
 			$item_normal
-			$sql_extra",
+			$approvals
+			$sql_extra LIMIT 100",
 			intval(self::$uid),
 			dbesc(self::$ob_hash)
 		);
 
 		if($r)
-			$result['network']['count'] = intval($r[0]['total']);
+			$result['network']['count'] = count($r);
 
 		return $result;
 	}
@@ -253,6 +258,9 @@ class Sse_bs extends Controller {
 
 		$item_normal = item_normal();
 
+		// FEP-5624 filter approvals for comments
+		$approvals = " AND verb NOT IN ('" . dbesc(ACTIVITY_ATTEND) . "', 'Accept', '" . dbesc(ACTIVITY_ATTENDNO) . "', 'Reject') ";
+
 		if ($notifications) {
 			$items = q("SELECT * FROM item
 				WHERE uid = %d
@@ -261,6 +269,7 @@ class Sse_bs extends Controller {
 				AND obj_type NOT IN ('Document', 'Video', 'Audio', 'Image')
 				AND author_xchan != '%s'
 				$item_normal
+				$approvals
 				$sql_extra
 				$sql_extra2
 				ORDER BY created DESC LIMIT $limit OFFSET $offset",
@@ -285,17 +294,18 @@ class Sse_bs extends Controller {
 
 		}
 
-		$r = q("SELECT count(id) as total FROM item
+		$r = q("SELECT id FROM item
 			WHERE uid = %d and item_unseen = 1 AND item_private = 2
 			$item_normal
+			$approvals
 			$sql_extra
-			AND author_xchan != '%s'",
+			AND author_xchan != '%s' LIMIT 100",
 			intval(self::$uid),
 			dbesc(self::$ob_hash)
 		);
 
 		if($r)
-			$result['dm']['count'] = intval($r[0]['total']);
+			$result['dm']['count'] = count($r);
 
 		return $result;
 	}
@@ -333,6 +343,9 @@ class Sse_bs extends Controller {
 
 		$item_normal = item_normal();
 
+		// FEP-5624 filter approvals for comments
+		$approvals = " AND verb NOT IN ('" . dbesc(ACTIVITY_ATTEND) . "', 'Accept', '" . dbesc(ACTIVITY_ATTENDNO) . "', 'Reject') ";
+
 		if ($notifications) {
 			$items = q("SELECT * FROM item
 				WHERE uid = %d
@@ -341,6 +354,7 @@ class Sse_bs extends Controller {
 				AND obj_type NOT IN ('Document', 'Video', 'Audio', 'Image')
 				AND author_xchan != '%s'
 				$item_normal
+				$approvals
 				$sql_extra
 				$sql_extra2
 				ORDER BY created DESC LIMIT $limit OFFSET $offset",
@@ -365,17 +379,18 @@ class Sse_bs extends Controller {
 
 		}
 
-		$r = q("SELECT count(id) as total FROM item
+		$r = q("SELECT id FROM item
 			WHERE uid = %d and item_unseen = 1 AND item_wall = 1 AND item_private IN (0, 1)
 			$item_normal
+			$approvals
 			$sql_extra
-			AND author_xchan != '%s'",
+			AND author_xchan != '%s' LIMIT 100",
 			intval(self::$uid),
 			dbesc(self::$ob_hash)
 		);
 
 		if($r)
-			$result['home']['count'] = intval($r[0]['total']);
+			$result['home']['count'] = count($r);
 
 		return $result;
 	}
@@ -421,21 +436,30 @@ class Sse_bs extends Controller {
 		if(self::$xchans)
 			$sql_extra2 = " AND CASE WHEN verb = '" . ACTIVITY_SHARE . "' THEN owner_xchan ELSE author_xchan END IN (" . self::$xchans . ") ";
 
+		$uids = " AND uid IN ( " . $sys['channel_id'] . " ) ";
+
+		$site_firehose = get_config('system', 'site_firehose', 0);
+		if($site_firehose) {
+			$uids = " AND uid IN ( " . stream_perms_api_uids(PERMS_PUBLIC) . " ) AND item_private = 0 AND item_wall = 1 ";
+		}
+
 		$item_normal = item_normal();
 
+		// FEP-5624 filter approvals for comments
+		$approvals = " AND verb NOT IN ('" . dbesc(ACTIVITY_ATTEND) . "', 'Accept', '" . dbesc(ACTIVITY_ATTENDNO) . "', 'Reject') ";
+
 		if ($notifications) {
 			$items = q("SELECT * FROM item
-				WHERE uid = %d
+				WHERE true $uids
 				AND created <= '%s'
-				AND item_unseen = 1
 				AND obj_type NOT IN ('Document', 'Video', 'Audio', 'Image')
 				AND author_xchan != '%s'
 				AND created > '%s'
 				$item_normal
+				$approvals
 				$sql_extra
 				$sql_extra2
 				ORDER BY created DESC LIMIT $limit OFFSET $offset",
-				intval($sys['channel_id']),
 				dbescdate($_SESSION['sse_loadtime']),
 				dbesc(self::$ob_hash),
 				dbescdate($_SESSION['static_loadtime'])
@@ -454,23 +478,21 @@ class Sse_bs extends Controller {
 			else {
 				$result['pubs']['offset'] = -1;
 			}
-
-
 		}
 
-		$r = q("SELECT count(id) as total FROM item
-			WHERE uid = %d AND item_unseen = 1
+		$r = q("SELECT id FROM item
+			WHERE true $uids
 			AND created > '%s'
 			$item_normal
+			$approvals
 			$sql_extra
-			AND author_xchan != '%s'",
-			intval($sys['channel_id']),
+			AND author_xchan != '%s' LIMIT 100",
 			dbescdate($_SESSION['static_loadtime']),
 			dbesc(self::$ob_hash)
 		);
 
 		if($r)
-			$result['pubs']['count'] = intval($r[0]['total']);
+			$result['pubs']['count'] = count($r);
 
 		return $result;
 	}

Zotlabs/Module/Totp_check.php

@@ -0,0 +1,86 @@
+<?php
+
+namespace Zotlabs\Module;
+
+use App;
+use Zotlabs\Web\Controller;
+use OTPHP\TOTP;
+
+class Totp_check extends Controller {
+
+	public function post() {
+		$retval = ['status' => false];
+		$static = $_POST['totp_code_static'] ?? false;
+
+		if (!local_channel()) {
+			if ($static) {
+				goaway(z_root());
+			}
+
+			json_return_and_die($retval);
+		}
+
+		$account = App::get_account();
+		if (!$account) {
+			json_return_and_die($retval);
+		}
+
+		$secret = $account['account_external'];
+		$input = (isset($_POST['totp_code'])) ? trim($_POST['totp_code']) : '';
+
+		if ($secret && $input) {
+			$otp = TOTP::create($secret); // create TOTP object from the secret.
+			if ($otp->verify($_POST['totp_code']) || $input === $secret ) {
+				logger('otp_success');
+				$_SESSION['2FA_VERIFIED'] = true;
+
+				if ($static) {
+					goaway(z_root());
+				}
+
+				$retval['status'] = true;
+				json_return_and_die($retval);
+			}
+			logger('otp_fail');
+		}
+
+		if ($static) {
+			if(empty($_SESSION['totp_try_count'])) {
+				$_SESSION['totp_try_count'] = 1;
+			}
+
+			if ($_SESSION['totp_try_count'] > 2) {
+				goaway('logout');
+			}
+
+			$_SESSION['totp_try_count']++;
+			goaway(z_root());
+		}
+
+		json_return_and_die($retval);
+	}
+
+	public function get() {
+
+		if (!local_channel() || App::$module === 'totp_check') {
+			goaway(z_root());
+		}
+
+		$account = App::get_account();
+		if (!$account) {
+			return t('Account not found.');
+		}
+
+		$id = $account['account_email'];
+
+		return replace_macros(get_markup_template('totp.tpl'),
+			[
+				'$header' => t('Multifactor Verification'),
+				'$id' => $id,
+				'$desc'   => t('Please enter the verification key from your authenticator app'),
+				'$submit' => t('Verify')
+			]
+		);
+	}
+}
+

Zotlabs/Module/Vote.php

@@ -45,9 +45,7 @@ class Vote extends Controller {
 
 		if ($obj['oneOf']) {
 			foreach($obj['oneOf'] as $selection) {
-				//		logger('selection: ' . $selection);
-				//		logger('response: ' . $response);
-				if($selection['name'] && $selection['name'] === $response) {
+				if($selection['name'] && htmlspecialchars_decode($selection['name']) === $response) {
 					$valid = true;
 				}
 			}
@@ -56,7 +54,7 @@ class Vote extends Controller {
 		$choices = [];
 		if ($obj['anyOf']) {
 			foreach ($obj['anyOf'] as $selection) {
-				$choices[] = $selection['name'];
+				$choices[] = htmlspecialchars_decode($selection['name']);
 			}
 			foreach ($response as $res) {
 				if (! in_array($res,$choices)) {

Zotlabs/Module/Wfinger.php

@@ -76,7 +76,10 @@ class Wfinger extends \Zotlabs\Web\Controller {
 				}
 			}
 			else {
-				$r = channelx_by_nick($channel);
+				// Also provide already deleted channels info here.
+				// This is required in the case where we need to verify keys
+				// of updates which we have got via directory sync.
+				$r = channelx_by_nick($channel, true);
 			}
 		}
 
@@ -94,10 +97,6 @@ class Wfinger extends \Zotlabs\Web\Controller {
 					'href' => z_root() . '/owa',
 				],
 			];
-
-
-
-
 		}
 
 		if($resource && $r) {

Zotlabs/Thumbs/Epubthumb.php

@@ -30,9 +30,14 @@ class Epubthumb {
 	 */
 	function Thumb($attach, $preview_style, $height = 300, $width = 300) {
 
+		$file = dbunescbin($attach['content']);
+		if (!$file) {
+			return;
+		}
+
 		$photo = false;
 
-		$ep = new \EPub(dbunescbin($attach['content']));
+		$ep = new \EPub($file);
 		$data = $ep->Cover();
 
 		if($data['found']) {

Zotlabs/Thumbs/Mp3audio.php

@@ -11,9 +11,16 @@ class Mp3audio {
 	}
 
 	function Thumb($attach,$preview_style,$height = 300, $width = 300) {
+
+		$file = dbunescbin($attach['content']);
+		if (!$file) {
+			return;
+		}
+
+		$photo = false;
 		$p = new ID3Parser();
 
-        $id = $p->analyze(dbunescbin($attach['content']));
+        $id = $p->analyze($file);
 
         $photo = isset($id['id3v2']['APIC'][0]['data']) ? $id['id3v2']['APIC'][0]['data'] : null;
         if(is_null($photo) && isset($id['id3v2']['PIC'][0]['data'])) {

Zotlabs/Thumbs/Pdf.php

@@ -11,9 +11,13 @@ class Pdf {
 
 	function Thumb($attach,$preview_style,$height = 300, $width = 300) {
 
+		$file = dbunescbin($attach['content']);
+		if (!$file) {
+			return;
+		}
+
 		$photo = false;
 
-		$file = dbunescbin($attach['content']);
 		$tmpfile = $file . '.pdf';
 		$outfile = $file . '.jpg';
 

Zotlabs/Thumbs/Text.php

@@ -11,7 +11,12 @@ class Text {
 
 	function Thumb($attach,$preview_style,$height = 300, $width = 300) {
 
-		$stream = @fopen(dbunescbin($attach['content']),'rb');
+		$file = dbunescbin($attach['content']);
+		if (!$file) {
+			return;
+		}
+
+		$stream = @fopen($file,'rb');
 		if($stream) {
 			$content = trim(stream_get_contents($stream,4096));
 			$content = str_replace("\r",'',$content);
@@ -25,13 +30,13 @@ class Text {
 			$colour = imagecolorallocate($image,0,0,0);
 			$border = imagecolorallocate($image,208,208,208);
 
-			$x1 = 0; 
-			$y1 = 0; 
-			$x2 = ImageSX($image) - 1; 
-			$y2 = ImageSY($image) - 1; 
+			$x1 = 0;
+			$y1 = 0;
+			$x2 = ImageSX($image) - 1;
+			$y2 = ImageSY($image) - 1;
 
-			for($i = 0; $i < 2; $i++) { 
-				ImageRectangle($image, $x1++, $y1++, $x2--, $y2--, $border); 
+			for($i = 0; $i < 2; $i++) {
+				ImageRectangle($image, $x1++, $y1++, $x2--, $y2--, $border);
 			}
 
 			foreach($content_a as $l => $t) {
@@ -46,4 +51,4 @@ class Text {
 			imagejpeg($image,dbunescbin($attach['content']) . '.thumb');
 		}
 	}
-}
+}

Zotlabs/Thumbs/Video.php

@@ -11,16 +11,19 @@ class Video {
 
 	function Thumb($attach,$preview_style,$height = 300, $width = 300) {
 
+		$file = dbunescbin($attach['content']);
+		if (!$file) {
+			return;
+		}
+
 		$photo = false;
 
 		$t = explode('/',$attach['filetype']);
 		if($t[1])
 			$extension = '.' . $t[1];
 		else
-			return; 
+			return;
 
-
-		$file = dbunescbin($attach['content']);
 		$tmpfile = $file . $extension;
 		$outfile = $file . '.jpg';
 
@@ -40,7 +43,7 @@ class Video {
 
 
 		$ffmpeg = trim(shell_exec('which ffmpeg'));
-		if($ffmpeg) { 
+		if($ffmpeg) {
 			logger('ffmpeg not found in path. Video thumbnails may fail.');
 		}
 
@@ -59,7 +62,7 @@ class Video {
 				@rename($outfile,$file . '.thumb');
 			}
 		}
-	
+
 		@unlink($tmpfile);
 	}
 }

Zotlabs/Update/_1254.php

@@ -0,0 +1,55 @@
+<?php
+
+namespace Zotlabs\Update;
+
+class _1254 {
+
+	function run() {
+
+		dbq("START TRANSACTION");
+
+		if(ACTIVE_DBTYPE == DBTYPE_POSTGRES) {
+			$r1 = dbq("CREATE TABLE IF NOT EXISTS workerq (
+				workerq_id bigserial NOT NULL,
+				workerq_priority smallint,
+				workerq_reservationid varchar(25) DEFAULT NULL,
+				workerq_processtimeout timestamp NOT NULL DEFAULT '0001-01-01 00:00:00',
+				workerq_data text,
+				workerq_uuid UUID NOT NULL,
+				PRIMARY KEY (workerq_id))"
+			);
+
+			$r2 = dbq("CREATE INDEX idx_workerq_priority ON workerq (workerq_priority)");
+			$r3 = dbq("CREATE INDEX idx_workerq_reservationid ON workerq (workerq_reservationid)");
+			$r4 = dbq("CREATE INDEX idx_workerq_processtimeout ON workerq (workerq_processtimeout)");
+			$r5 = dbq("CREATE INDEX idx_workerq_uuid ON workerq (workerq_uuid)");
+
+			$r = ($r1 && $r2 && $r3 && $r4 && $r5);
+		}
+		else {
+			$r = dbq("CREATE TABLE IF NOT EXISTS workerq (
+				workerq_id BIGINT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+				workerq_priority smallint,
+				workerq_reservationid varchar(25) DEFAULT NULL,
+				workerq_processtimeout datetime NOT NULL DEFAULT '0001-01-01 00:00:00',
+				workerq_data text,
+				workerq_uuid char(36) NOT NULL DEFAULT '',
+				KEY workerq_priority (workerq_priority),
+				KEY workerq_reservationid (workerq_reservationid),
+				KEY workerq_processtimeout (workerq_uuid),
+				KEY workerq_uuid (workerq_processtimeout)
+				) ENGINE = InnoDB DEFAULT CHARSET=utf8mb4"
+			);
+		}
+
+		if($r) {
+			dbq("COMMIT");
+			return UPDATE_SUCCESS;
+		}
+
+		q("ROLLBACK");
+		return UPDATE_FAILED;
+
+	}
+
+}

Zotlabs/Update/_1255.php

@@ -0,0 +1,29 @@
+<?php
+
+namespace Zotlabs\Update;
+
+class _1255 {
+
+	function run() {
+
+		dbq("START TRANSACTION");
+
+		if(ACTIVE_DBTYPE == DBTYPE_POSTGRES) {
+			$r = q("ALTER TABLE workerq add workerq_cmd text NOT NULL DEFAULT ''");
+		}
+
+		if(ACTIVE_DBTYPE == DBTYPE_MYSQL) {
+			$r = q("ALTER TABLE workerq add workerq_cmd varchar(191) NOT NULL DEFAULT ''");
+		}
+
+		if($r) {
+			dbq("COMMIT");
+			return UPDATE_SUCCESS;
+		}
+
+		q("ROLLBACK");
+		return UPDATE_FAILED;
+
+	}
+
+}

Zotlabs/Update/_1256.php

@@ -0,0 +1,36 @@
+<?php
+
+namespace Zotlabs\Update;
+
+class _1256 {
+
+	function run() {
+
+		dbq("START TRANSACTION");
+
+		$r1 = dbq("TRUNCATE TABLE updates");
+
+		if(ACTIVE_DBTYPE == DBTYPE_POSTGRES) {
+			$r2a = dbq("ALTER TABLE updates add ud_update numeric(1) NOT NULL DEFAULT '0'");
+			$r2b = dbq("CREATE INDEX ud_update ON updates (ud_update)");
+
+			$r2 = ($r2a && $r2b);
+		}
+
+		if(ACTIVE_DBTYPE == DBTYPE_MYSQL) {
+			$r2 = dbq("ALTER TABLE updates add ud_update tinyint(1) NOT NULL DEFAULT '0',
+				ADD INDEX (ud_update);"
+			);
+		}
+
+		if($r1 && $r2) {
+			dbq("COMMIT");
+			return UPDATE_SUCCESS;
+		}
+
+		q("ROLLBACK");
+		return UPDATE_FAILED;
+
+	}
+
+}

Zotlabs/Update/_1257.php

@@ -0,0 +1,33 @@
+<?php
+
+namespace Zotlabs\Update;
+
+class _1257 {
+
+	function run() {
+
+		dbq("START TRANSACTION");
+
+		$r1 = dbq("TRUNCATE TABLE updates");
+
+		if(ACTIVE_DBTYPE == DBTYPE_POSTGRES) {
+			$r2 = dbq("ALTER TABLE updates add ud_host text NOT NULL DEFAULT ''");
+		}
+
+		if(ACTIVE_DBTYPE == DBTYPE_MYSQL) {
+			$r2 = dbq("ALTER TABLE updates add ud_host varchar(191) NOT NULL DEFAULT ''");
+		}
+
+		$r3 = dbq("ALTER TABLE updates DROP COLUMN ud_guid");
+
+		if($r1 && $r2 && $r3) {
+			dbq("COMMIT");
+			return UPDATE_SUCCESS;
+		}
+
+		q("ROLLBACK");
+		return UPDATE_FAILED;
+
+	}
+
+}

Zotlabs/Update/_1258.php

@@ -0,0 +1,14 @@
+<?php
+
+namespace Zotlabs\Update;
+
+class _1258 {
+
+	function run() {
+
+		dbq("DELETE FROM pconfig WHERE cat = 'channelreputation'");
+		dbq("DELETE FROM iconfig WHERE cat = 'channelreputation'");
+
+	}
+
+}

Zotlabs/Web/SessionRedis.php

@@ -42,7 +42,7 @@ class SessionRedis implements \SessionHandlerInterface {
                 }
         }
 
-
+        #[\ReturnTypeWillChange]
         function open($s, $n) {
 
                 return true;
@@ -53,6 +53,7 @@ class SessionRedis implements \SessionHandlerInterface {
         // some which call read explicitly and some that do not. So we call it explicitly
         // just after sid regeneration to force a record to exist.
 
+        #[\ReturnTypeWillChange]
         function read($id) {
 
                 if ($id) {
@@ -67,7 +68,7 @@ class SessionRedis implements \SessionHandlerInterface {
                 return '';
         }
 
-
+        #[\ReturnTypeWillChange]
         function write($id, $data) {
 
                 // Pretend everything is hunky-dory, even though it isn't.
@@ -100,13 +101,13 @@ class SessionRedis implements \SessionHandlerInterface {
                 return true;
         }
 
-
+        #[\ReturnTypeWillChange]
         function close() {
 
                 return true;
         }
 
-
+        #[\ReturnTypeWillChange]
         function destroy ($id) {
 
                 $this->redis->del($id);
@@ -114,7 +115,7 @@ class SessionRedis implements \SessionHandlerInterface {
                 return true;
         }
 
-
+        #[\ReturnTypeWillChange]
         function gc($expire) {
 
                 return true;

Zotlabs/Web/WebServer.php

@@ -50,7 +50,8 @@ class WebServer {
 			else
 				unset($_SESSION['language']);
 		}
-		if((x($_SESSION, 'language')) && ($_SESSION['language'] !== $lang)) {
+
+		if ((x($_SESSION, 'language')) && ($_SESSION['language'] !== \App::$language)) {
 			\App::$language = $_SESSION['language'];
 			load_translation_table(\App::$language);
 		}

Zotlabs/Widget/Admin.php

@@ -32,6 +32,7 @@ class Admin {
 			'addons'    => array(z_root() . '/admin/addons/',   t('Addons'),         'addons'),
 			'themes'    => array(z_root() . '/admin/themes/',   t('Themes'),         'themes'),
 			'queue'     => array(z_root() . '/admin/queue',     t('Inspect queue'),  'queue'),
+			'queueworker' => array(z_root() . '/admin/queueworker', t('Queueworker'),  'queueworker'),
 			'profs'     => array(z_root() . '/admin/profs',     t('Profile Fields'), 'profs'),
 			'dbsync'    => array(z_root() . '/admin/dbsync/',   t('DB updates'),     'dbsync')
 		];

Zotlabs/Widget/Categories.php

@@ -3,7 +3,7 @@
 /**
  *   * Name: Categories
  *   * Description: Display a menu with links to categories
- *   * Requires: channel, articles, cards, cloud
+ *   * Requires: channel, cloud
  */
 
 namespace Zotlabs\Widget;
@@ -24,15 +24,16 @@ class Categories {
 		}
 
 		$cat = ((x($_REQUEST, 'cat')) ? htmlspecialchars($_REQUEST['cat'], ENT_COMPAT, 'UTF-8') : '');
-		$srchurl = App::$query_string;
-		$srchurl = rtrim(preg_replace('/cat\=[^\&].*?(\&|$)/is', '', $srchurl), '&');
-		$srchurl = str_replace(['?f=','&f=', '/?'], ['', '', ''], $srchurl);
+
+		// Discard queries from the current URL, as the template expects a base
+		// URL without any queries.
+		$base = substr(App::$query_string, 0, strcspn(App::$query_string, '?'));
 
 		if($files) {
-			return filecategories_widget($srchurl, $cat);
+			return filecategories_widget($base, $cat);
 		}
 
-		return categories_widget($srchurl, $cat);
+		return categories_widget($base, $cat);
 
 	}
 }

Zotlabs/Widget/Channel_activities.php

@@ -26,7 +26,7 @@ class Channel_activities {
 		self::$channel = App::get_channel();
 
 		$o = '<div id="channel-activities" class="d-none overflow-hidden">';
-		$o .= '<h2 class="mb-4">Welcome ' . self::$channel['channel_name'] . '!</h2>';
+		$o .= '<h2 class="mb-4">' . t('Welcome') . ' ' . self::$channel['channel_name'] . '!</h2>';
 		//$o .= 'Last login date: ' . get_pconfig(self::$uid, 'system', 'stored_login_date') . ' from ' . get_pconfig(self::$uid, 'system', 'stored_login_addr');
 
 		self::get_photos_activity();
@@ -43,7 +43,7 @@ class Channel_activities {
 		call_hooks('channel_activities_widget', $hookdata);
 
 		if (!$hookdata['activities']) {
-			$o .= '<h3>No recent activity to display</h3>';
+			$o .= '<h3>' . t('No recent activities') . '</h3>';
 			$o .= '</div>';
 			return $o;
 		}
@@ -179,7 +179,7 @@ class Channel_activities {
 
 		$account = App::get_account();
 
-		$r = q("SELECT channel_id, channel_name, xchan_photo_s FROM channel
+		$r = q("SELECT channel_id, channel_name, xchan_addr, xchan_photo_s FROM channel
 			LEFT JOIN xchan ON channel_hash = xchan_hash
 			WHERE channel_account_id = %d
 			AND channel_id != %d AND channel_removed = 0",
@@ -223,7 +223,7 @@ class Channel_activities {
 			$i[] = [
 				'url' => z_root() . '/manage/' . $rr['channel_id'],
 				'title' => '',
-				'summary' => '<img src="' . $rr['xchan_photo_s'] . '" class="menu-img-2">' . $rr['channel_name'],
+				'summary' => '<div class="text-truncate lh-sm"><img src="' . $rr['xchan_photo_s'] . '" class="menu-img-2">' . '<strong>' . $rr['channel_name'] . '</strong><br><small class="opacity-75">' . $rr['xchan_addr'] . '</small></div>',
 				'footer' => $footer
 			];
 

Zotlabs/Widget/Messages.php

@@ -34,7 +34,9 @@ class Messages {
 				'starred_messages_title' => t('Starred messages'),
 				'notice_messages_title' => t('Notices'),
 				'loading' => t('Loading'),
-				'empty' => t('No messages')
+				'empty' => t('No messages'),
+				'unseen_count' => t('Unseen'),
+				'filter' => t('Filter by name or address')
 			]
 		]);
 
@@ -46,7 +48,8 @@ class Messages {
 			return;
 
 		$offset = $options['offset'] ?? 0;
-		$type = $options['type'] ?? 'default';
+		$type = $options['type'] ?? '';
+		$author = $options['author'] ?? '';
 
 		if ($offset == -1) {
 			return;
@@ -57,34 +60,60 @@ class Messages {
 		}
 
 		$channel = App::get_channel();
-		$item_normal = item_normal();
+		$item_normal_i = str_replace('item.', 'i.', item_normal());
+		$item_normal_c = str_replace('item.', 'c.', item_normal());
 		$entries = [];
 		$limit = 30;
 		$dummy_order_sql = '';
+		$author_sql = '';
 		$loadtime = (($offset) ? $_SESSION['messages_loadtime'] : datetime_convert());
+		$vnotify = get_pconfig(local_channel(), 'system', 'vnotify', -1);
+
+		$vnotify_sql_c = '';
+		$vnotify_sql_i = '';
+
+		if (!($vnotify & VNOTIFY_LIKE)) {
+			$vnotify_sql_c = " AND c.verb NOT IN ('" . dbesc(ACTIVITY_LIKE) . "', '" . dbesc(ACTIVITY_DISLIKE) . "') ";
+			$vnotify_sql_i = " AND i.verb NOT IN ('" . dbesc(ACTIVITY_LIKE) . "', '" . dbesc(ACTIVITY_DISLIKE) . "') ";
+		}
+		elseif (!feature_enabled(local_channel(), 'dislike')) {
+			$vnotify_sql_c = " AND c.verb NOT IN ('" . dbesc(ACTIVITY_DISLIKE) . "') ";
+			$vnotify_sql_i = " AND i.verb NOT IN ('" . dbesc(ACTIVITY_DISLIKE) . "') ";
+		}
+
+		if($author) {
+			$author_sql = " AND i.owner_xchan = '" . protect_sprintf(dbesc($author)) . "' ";
+		}
 
 		switch($type) {
 			case 'direct':
-				$type_sql = ' AND item_private = 2 ';
+				$type_sql = ' AND i.item_private = 2 ';
 				// $dummy_order_sql has no other meaning but to trick
 				// some mysql backends into using the right index.
-				$dummy_order_sql = ', received DESC ';
+				$dummy_order_sql = ', i.received DESC ';
 				break;
 			case 'starred':
-				$type_sql = ' AND item_starred = 1 ';
+				$type_sql = ' AND i.item_starred = 1 ';
 				break;
 			default:
-				$type_sql = ' AND item_private IN (0, 1) ';
+				$type_sql = ' AND i.item_private IN (0, 1) ';
 		}
 
-		$items = q("SELECT * FROM item WHERE uid = %d
-			AND created <= '%s'
+		// FEP-5624 filter approvals for comments
+		$approvals_c = " AND c.verb NOT IN ('" . dbesc(ACTIVITY_ATTEND) . "', 'Accept', '" . dbesc(ACTIVITY_ATTENDNO) . "', 'Reject') ";
+
+		$items = q("SELECT *,
+			(SELECT count(*) FROM item c WHERE c.uid = %d AND c.parent = i.parent AND c.item_unseen = 1 AND c.item_thread_top = 0 $item_normal_c $approvals_c $vnotify_sql_c) AS unseen_count
+			FROM item i WHERE i.uid = %d
+			AND i.created <= '%s'
 			$type_sql
-			AND item_thread_top = 1
-			$item_normal
-			ORDER BY created DESC $dummy_order_sql
+			AND i.item_thread_top = 1
+			$author_sql
+			$item_normal_i
+			ORDER BY i.created DESC $dummy_order_sql
 			LIMIT $limit OFFSET $offset",
 			intval(local_channel()),
+			intval(local_channel()),
 			dbescdate($loadtime)
 		);
 
@@ -133,7 +162,7 @@ class Messages {
 				$summary = '...';
 			}
 			else {
-				$summary = substr_words($summary, 68);
+				$summary = substr_words($summary, 140);
 			}
 
 			switch(intval($item['item_private'])) {
@@ -148,13 +177,16 @@ class Messages {
 			}
 
 			$entries[$i]['author_name'] = $item['author']['xchan_name'];
-			$entries[$i]['author_addr'] = $item['author']['xchan_addr'] ?? $item['author']['xchan_url'];
+			$entries[$i]['author_addr'] = (($item['author']['xchan_addr']) ? $item['author']['xchan_addr'] : $item['author']['xchan_url']);
+			$entries[$i]['author_img'] = $item['author']['xchan_photo_s'];
 			$entries[$i]['info'] = $info;
 			$entries[$i]['created'] = datetime_convert('UTC', date_default_timezone_get(), $item['created']);
 			$entries[$i]['summary'] = $summary;
 			$entries[$i]['b64mid'] = gen_link_id($item['mid']);
 			$entries[$i]['href'] = z_root() . '/hq/' . gen_link_id($item['mid']);
 			$entries[$i]['icon'] = $icon;
+			$entries[$i]['unseen_count'] = (($item['unseen_count']) ? $item['unseen_count'] : (($item['item_unseen']) ? '&#8192;' : ''));
+			$entries[$i]['unseen_class'] = (($item['item_unseen']) ? 'primary' : 'secondary');
 
 			$i++;
 		}
@@ -218,6 +250,7 @@ class Messages {
 		if (!local_channel())
 			return;
 
+
 		$limit  = 30;
 
 		$offset = 0;
@@ -225,7 +258,14 @@ class Messages {
 			$offset = intval($options['offset']);
 		}
 
-		$notices = q("SELECT * FROM notify WHERE uid = %d
+		$author_url = $options['author'] ?? '';
+		$author_sql = '';
+
+		if($author_url) {
+			$author_sql = " AND url = '" . protect_sprintf(dbesc($author_url)) . "' ";
+		}
+
+		$notices = q("SELECT * FROM notify WHERE uid = %d $author_sql
 			ORDER BY created DESC LIMIT $limit OFFSET $offset",
 			intval(local_channel())
 		);
@@ -243,6 +283,7 @@ class Messages {
 
 			$entries[$i]['author_name'] = $notice['xname'];
 			$entries[$i]['author_addr'] = $notice['url'];
+			$entries[$i]['author_img'] = $notice['photo'];// $item['author']['xchan_photo_s'];
 			$entries[$i]['info'] = '';
 			$entries[$i]['created'] = datetime_convert('UTC', date_default_timezone_get(), $notice['created']);
 			$entries[$i]['summary'] = $summary;

Zotlabs/Widget/Pinned.php

@@ -42,6 +42,8 @@ class Pinned {
 
 		$observer = \App::get_observer();
 
+		xchan_query($items);
+
 		foreach($items as $item) {
 
 			$midb64 = gen_link_id($item['mid']);
@@ -49,8 +51,8 @@ class Pinned {
 			if(isset($observer['xchan_hash']) && in_array($observer['xchan_hash'], get_pconfig($item['uid'], 'pinned_hide', $midb64, [])))
 				continue;
 
-			$author = channelx_by_hash($item['author_xchan']);
-			$owner = channelx_by_hash($item['owner_xchan']);
+			$author = $item['author'];
+			$owner = $item['owner'];
 
 			$profile_avatar = $author['xchan_photo_m'];
 			$profile_link = chanlink_hash($item['author_xchan']);
@@ -65,7 +67,7 @@ class Pinned {
 
 			$conv_responses = [];
 
-			if($item['obj_type'] === ACTIVITY_OBJ_EVENT) {
+			if(in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT])) {
 				$conv_responses['attendyes'] = [ 'title' => t('Attending','title') ];
 				$conv_responses['attendno'] = [ 'title' => t('Not attending','title') ];
 				$conv_responses['attendmaybe'] = [ 'title' => t('Might attend','title') ];
@@ -159,7 +161,9 @@ class Pinned {
 				'hide'		 => (! $is_new && isset($observer['xchan_hash']) && $observer['xchan_hash'] != $owner['xchan_hash'] ? t("Don't show") : ''),
 				// end toolbar buttons
 				'modal_dismiss' => t('Close'),
-				'responses'	 => $conv_responses
+				'responses'	 => $conv_responses,
+				'author_id' => (($author['xchan_addr']) ? $author['xchan_addr'] : $author['xchan_url'])
+
 			];
 
 			$tpl = get_markup_template('pinned_item.tpl');

Zotlabs/Widget/Pubtagcloud.php

@@ -22,24 +22,19 @@ class Pubtagcloud {
             }
         }
 
-        $site_firehose = ((intval(get_config('system','site_firehose',0))) ? true : false);
-        $net_firehose  = ((get_config('system','disable_discover_tab',1)) ? false : true);
+		$net_firehose  = ((get_config('system','disable_discover_tab',1)) ? false : true);
 
-        if(! ($site_firehose || $net_firehose)) {
-            return EMPTY_STR;
-        }
+		if(!$net_firehose) {
+			return '';
+		}
 
-        if($net_firehose) {
-            $site_firehose = false;
-        }
+		$site_firehose = ((intval(get_config('system','site_firehose',0))) ? true : false);
 
 		$safemode = get_xconfig(get_observer_hash(),'directory','safemode',1);
 
-
-
 		$limit = ((array_key_exists('limit', $arr)) ? intval($arr['limit']) : 75);
 
-		return pubtagblock($net_firehose,$site_firehose, $limit, $trending, $safemode);
+		return pubtagblock($net_firehose, $site_firehose, $limit, $trending, $safemode);
 
 		return '';
 	}

boot.php

@@ -60,10 +60,10 @@ require_once('include/bbcode.php');
 require_once('include/items.php');
 
 define('PLATFORM_NAME', 'hubzilla');
-define('STD_VERSION', '7.8.4');
+define('STD_VERSION', '8.4');
 define('ZOT_REVISION', '6.0');
 
-define('DB_UPDATE_VERSION', 1253);
+define('DB_UPDATE_VERSION', 1258);
 
 define('PROJECT_BASE', __DIR__);
 
@@ -84,6 +84,10 @@ define('DIRECTORY_MODE_PRIMARY', 0x0001); // There can only be *one* primary dir
 define('DIRECTORY_MODE_SECONDARY', 0x0002); // All other mirror directory servers
 define('DIRECTORY_MODE_STANDALONE', 0x0100); // A detached (off the grid) hub with itself as directory server.
 
+define('DIRECTORY_FLAG_OK',     0);
+define('DIRECTORY_FLAG_UNSAFE', 1);
+define('DIRECTORY_FLAG_HIDDEN', 2);
+
 // We will look for upstream directories whenever me make contact
 // with other sites, but if this is a new installation and isn't
 // a standalone hub, we need to seed the service with a starting
@@ -96,8 +100,7 @@ define('DIRECTORY_FALLBACK_MASTER', 'https://hub.netzgemeinde.eu');
 function get_directory_fallback_servers() {
 	$ret = [
 		'https://hub.netzgemeinde.eu',
-		'https://zotsite.net',
-		'https://hub.libranet.de'
+		'https://zotsite.net'
 	];
 
 	return $ret;
@@ -364,11 +367,6 @@ define('POLL_TENSCALE', 0x0002);  // 1-10
 define('POLL_MULTIPLE_CHOICE', 0x0004);
 define('POLL_OVERWRITE', 0x8000);  // If you vote twice remove the prior entry
 
-
-define('UPDATE_FLAGS_UPDATED', 0x0001);
-define('UPDATE_FLAGS_FORCED', 0x0002);
-define('UPDATE_FLAGS_DELETED', 0x1000);
-
 define('HUBLOC_OFFLINE', 0x0001);
 
 define('DROPITEM_NORMAL', 0);
@@ -1862,10 +1860,9 @@ function can_view_public_stream() {
 		}
 	}
 
-	$site_firehose = ((intval(get_config('system', 'site_firehose', 0))) ? true : false);
 	$net_firehose  = ((get_config('system', 'disable_discover_tab', 1)) ? false : true);
 
-	if (!($site_firehose || $net_firehose)) {
+	if (!$net_firehose) {
 		return false;
 	}
 
@@ -2045,24 +2042,8 @@ function proc_run() {
 	if (!$arr['run_cmd'])
 		return;
 
-	if (count($args) && $args[0] === 'php')
+	if (count($args) && $args[0] === 'php') {
 		$args[0] = ((x(App::$config, 'system')) && (x(App::$config['system'], 'php_path')) && (strlen(App::$config['system']['php_path'])) ? App::$config['system']['php_path'] : 'php');
-
-
-	// redirect proc_run statements of legacy daemon processes to the newer Daemon Master object class
-	// We will keep this interface until everybody has transitioned. (2016-05-20)
-
-	if (strstr($args[1], 'include/')) {
-		// convert 'include/foo.php' to 'Foo'
-		$orig = substr(ucfirst(substr($args[1], 8)), 0, -4);
-		logger('proc_run_redirect: ' . $orig);
-		if (file_exists('Zotlabs/Daemon/' . $orig . '.php')) {
-			array_shift($args); // daemons are all run by php, pop it off the top of the array
-			$args[0] = $orig;   // replace with the new daemon name
-			logger('Redirecting old proc_run interface: ' . print_r($args, true), LOGGER_DEBUG, LOG_DEBUG);
-			Master::Summon($args); // summon the daemon
-			return;
-		}
 	}
 
 	$args    = array_map('escapeshellarg', $args);
@@ -2284,6 +2265,7 @@ function load_pdl() {
 			'module' => App::$module,
 			'layout' => ''
 		];
+
 		/**
 		 * @hooks load_pdl
 		 *   Called when we load a PDL file or description.
@@ -2297,20 +2279,26 @@ function load_pdl() {
 		$u = App::$comanche->get_channel_id();
 		$s = '';
 
-		if ($u)
+		if ($u) {
 			$s = get_pconfig($u, 'system', $n);
-		if (!$s)
-			$s = $layout;
+		}
 
-		if ((!$s) && (($p = theme_include($n)) != ''))
+		if (!$s) {
+			$s = $layout;
+		}
+
+		if ((!$s) && (($p = theme_include($n)) != '')) {
 			$s = @file_get_contents($p);
-		elseif (file_exists('addon/' . App::$module . '/' . $n))
+		}
+		elseif ((!$s) && file_exists('addon/' . App::$module . '/' . $n)) {
 			$s = @file_get_contents('addon/' . App::$module . '/' . $n);
+		}
 
 		$arr = [
 			'module' => App::$module,
 			'layout' => $s
 		];
+
 		call_hooks('alter_pdl', $arr);
 		$s = $arr['layout'];
 

composer.json

@@ -41,11 +41,13 @@
         "lukasreschke/id3parser": "^0.0.3",
         "smarty/smarty": "^4.1",
         "ramsey/uuid": "^4.1",
-        "twbs/bootstrap": "^5.0.1",
+        "twbs/bootstrap": "5.3.0-alpha3",
         "blueimp/jquery-file-upload": "^10.3",
         "desandro/imagesloaded": "^4.1",
         "phpseclib/phpseclib": "~2.0",
-        "jbroadway/urlify": "^1.2"
+        "jbroadway/urlify": "^1.2",
+        "chillerlan/php-qrcode": "^4.3",
+        "spomky-labs/otphp": "^11.1"
     },
     "require-dev": {
         "phpunit/phpunit": "^9.4",

composer.lock

@@ -4,7 +4,7 @@
         "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
         "This file is @generated automatically"
     ],
-    "content-hash": "b0942189dde4fd59450e03c6eae70758",
+    "content-hash": "3bb482c195c6645e16e005862dbf09d2",
     "packages": [
         {
             "name": "blueimp/jquery-file-upload",
@@ -194,6 +194,148 @@
             },
             "time": "2022-05-31T16:12:58+00:00"
         },
+        {
+            "name": "chillerlan/php-qrcode",
+            "version": "4.3.4",
+            "source": {
+                "type": "git",
+                "url": "https://github.com/chillerlan/php-qrcode.git",
+                "reference": "2ca4bf5ae048af1981d1023ee42a0a2a9d51e51d"
+            },
+            "dist": {
+                "type": "zip",
+                "url": "https://api.github.com/repos/chillerlan/php-qrcode/zipball/2ca4bf5ae048af1981d1023ee42a0a2a9d51e51d",
+                "reference": "2ca4bf5ae048af1981d1023ee42a0a2a9d51e51d",
+                "shasum": ""
+            },
+            "require": {
+                "chillerlan/php-settings-container": "^2.1.4",
+                "ext-mbstring": "*",
+                "php": "^7.4 || ^8.0"
+            },
+            "require-dev": {
+                "phan/phan": "^5.3",
+                "phpunit/phpunit": "^9.5",
+                "setasign/fpdf": "^1.8.2"
+            },
+            "suggest": {
+                "chillerlan/php-authenticator": "Yet another Google authenticator! Also creates URIs for mobile apps.",
+                "setasign/fpdf": "Required to use the QR FPDF output."
+            },
+            "type": "library",
+            "autoload": {
+                "psr-4": {
+                    "chillerlan\\QRCode\\": "src/"
+                }
+            },
+            "notification-url": "https://packagist.org/downloads/",
+            "license": [
+                "MIT"
+            ],
+            "authors": [
+                {
+                    "name": "Kazuhiko Arase",
+                    "homepage": "https://github.com/kazuhikoarase"
+                },
+                {
+                    "name": "Smiley",
+                    "email": "smiley@chillerlan.net",
+                    "homepage": "https://github.com/codemasher"
+                },
+                {
+                    "name": "Contributors",
+                    "homepage": "https://github.com/chillerlan/php-qrcode/graphs/contributors"
+                }
+            ],
+            "description": "A QR code generator. PHP 7.4+",
+            "homepage": "https://github.com/chillerlan/php-qrcode",
+            "keywords": [
+                "phpqrcode",
+                "qr",
+                "qr code",
+                "qrcode",
+                "qrcode-generator"
+            ],
+            "support": {
+                "issues": "https://github.com/chillerlan/php-qrcode/issues",
+                "source": "https://github.com/chillerlan/php-qrcode/tree/4.3.4"
+            },
+            "funding": [
+                {
+                    "url": "https://www.paypal.com/donate?hosted_button_id=WLYUNAT9ZTJZ4",
+                    "type": "custom"
+                },
+                {
+                    "url": "https://ko-fi.com/codemasher",
+                    "type": "ko_fi"
+                }
+            ],
+            "time": "2022-07-25T09:12:45+00:00"
+        },
+        {
+            "name": "chillerlan/php-settings-container",
+            "version": "2.1.4",
+            "source": {
+                "type": "git",
+                "url": "https://github.com/chillerlan/php-settings-container.git",
+                "reference": "1beb7df3c14346d4344b0b2e12f6f9a74feabd4a"
+            },
+            "dist": {
+                "type": "zip",
+                "url": "https://api.github.com/repos/chillerlan/php-settings-container/zipball/1beb7df3c14346d4344b0b2e12f6f9a74feabd4a",
+                "reference": "1beb7df3c14346d4344b0b2e12f6f9a74feabd4a",
+                "shasum": ""
+            },
+            "require": {
+                "ext-json": "*",
+                "php": "^7.4 || ^8.0"
+            },
+            "require-dev": {
+                "phan/phan": "^5.3",
+                "phpunit/phpunit": "^9.5"
+            },
+            "type": "library",
+            "autoload": {
+                "psr-4": {
+                    "chillerlan\\Settings\\": "src/"
+                }
+            },
+            "notification-url": "https://packagist.org/downloads/",
+            "license": [
+                "MIT"
+            ],
+            "authors": [
+                {
+                    "name": "Smiley",
+                    "email": "smiley@chillerlan.net",
+                    "homepage": "https://github.com/codemasher"
+                }
+            ],
+            "description": "A container class for immutable settings objects. Not a DI container. PHP 7.4+",
+            "homepage": "https://github.com/chillerlan/php-settings-container",
+            "keywords": [
+                "PHP7",
+                "Settings",
+                "configuration",
+                "container",
+                "helper"
+            ],
+            "support": {
+                "issues": "https://github.com/chillerlan/php-settings-container/issues",
+                "source": "https://github.com/chillerlan/php-settings-container"
+            },
+            "funding": [
+                {
+                    "url": "https://www.paypal.com/donate?hosted_button_id=WLYUNAT9ZTJZ4",
+                    "type": "custom"
+                },
+                {
+                    "url": "https://ko-fi.com/codemasher",
+                    "type": "ko_fi"
+                }
+            ],
+            "time": "2022-07-05T22:32:14+00:00"
+        },
         {
             "name": "commerceguys/intl",
             "version": "v1.1.1",
@@ -602,6 +744,73 @@
             },
             "time": "2022-09-26T12:21:08+00:00"
         },
+        {
+            "name": "paragonie/constant_time_encoding",
+            "version": "v2.6.3",
+            "source": {
+                "type": "git",
+                "url": "https://github.com/paragonie/constant_time_encoding.git",
+                "reference": "58c3f47f650c94ec05a151692652a868995d2938"
+            },
+            "dist": {
+                "type": "zip",
+                "url": "https://api.github.com/repos/paragonie/constant_time_encoding/zipball/58c3f47f650c94ec05a151692652a868995d2938",
+                "reference": "58c3f47f650c94ec05a151692652a868995d2938",
+                "shasum": ""
+            },
+            "require": {
+                "php": "^7|^8"
+            },
+            "require-dev": {
+                "phpunit/phpunit": "^6|^7|^8|^9",
+                "vimeo/psalm": "^1|^2|^3|^4"
+            },
+            "type": "library",
+            "autoload": {
+                "psr-4": {
+                    "ParagonIE\\ConstantTime\\": "src/"
+                }
+            },
+            "notification-url": "https://packagist.org/downloads/",
+            "license": [
+                "MIT"
+            ],
+            "authors": [
+                {
+                    "name": "Paragon Initiative Enterprises",
+                    "email": "security@paragonie.com",
+                    "homepage": "https://paragonie.com",
+                    "role": "Maintainer"
+                },
+                {
+                    "name": "Steve 'Sc00bz' Thomas",
+                    "email": "steve@tobtu.com",
+                    "homepage": "https://www.tobtu.com",
+                    "role": "Original Developer"
+                }
+            ],
+            "description": "Constant-time Implementations of RFC 4648 Encoding (Base-64, Base-32, Base-16)",
+            "keywords": [
+                "base16",
+                "base32",
+                "base32_decode",
+                "base32_encode",
+                "base64",
+                "base64_decode",
+                "base64_encode",
+                "bin2hex",
+                "encoding",
+                "hex",
+                "hex2bin",
+                "rfc4648"
+            ],
+            "support": {
+                "email": "info@paragonie.com",
+                "issues": "https://github.com/paragonie/constant_time_encoding/issues",
+                "source": "https://github.com/paragonie/constant_time_encoding"
+            },
+            "time": "2022-06-14T06:56:20+00:00"
+        },
         {
             "name": "pear/text_languagedetect",
             "version": "v1.0.1",
@@ -1429,26 +1638,27 @@
         },
         {
             "name": "simplepie/simplepie",
-            "version": "1.7.0",
+            "version": "1.8.0",
             "source": {
                 "type": "git",
                 "url": "https://github.com/simplepie/simplepie.git",
-                "reference": "9e9add3428ce86aede874bcf9a59c78e272f8dc1"
+                "reference": "65b095d87bc00898d8fa7737bdbcda93a3fbcc55"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/simplepie/simplepie/zipball/9e9add3428ce86aede874bcf9a59c78e272f8dc1",
-                "reference": "9e9add3428ce86aede874bcf9a59c78e272f8dc1",
+                "url": "https://api.github.com/repos/simplepie/simplepie/zipball/65b095d87bc00898d8fa7737bdbcda93a3fbcc55",
+                "reference": "65b095d87bc00898d8fa7737bdbcda93a3fbcc55",
                 "shasum": ""
             },
             "require": {
                 "ext-pcre": "*",
                 "ext-xml": "*",
                 "ext-xmlreader": "*",
-                "php": ">=5.6.0"
+                "php": ">=7.2.0"
             },
             "require-dev": {
                 "friendsofphp/php-cs-fixer": "^2.19 || ^3.8",
+                "psr/simple-cache": "^1 || ^2 || ^3",
                 "yoast/phpunit-polyfills": "^1.0.1"
             },
             "suggest": {
@@ -1507,23 +1717,23 @@
                 "rss"
             ],
             "support": {
-                "source": "https://github.com/simplepie/simplepie/tree/1.7.0",
+                "source": "https://github.com/simplepie/simplepie/tree/1.8.0",
                 "issues": "https://github.com/simplepie/simplepie/issues"
             },
-            "time": "2022-09-30T06:49:48+00:00"
+            "time": "2023-01-20T08:37:35+00:00"
         },
         {
             "name": "smarty/smarty",
-            "version": "v4.2.1",
+            "version": "v4.3.0",
             "source": {
                 "type": "git",
                 "url": "https://github.com/smarty-php/smarty.git",
-                "reference": "ffa2b81a8e354a49fd8a2f24742dc9dc399e8007"
+                "reference": "c02e9e135ea719b91f457a0072748ded0e852e7d"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/smarty-php/smarty/zipball/ffa2b81a8e354a49fd8a2f24742dc9dc399e8007",
-                "reference": "ffa2b81a8e354a49fd8a2f24742dc9dc399e8007",
+                "url": "https://api.github.com/repos/smarty-php/smarty/zipball/c02e9e135ea719b91f457a0072748ded0e852e7d",
+                "reference": "c02e9e135ea719b91f457a0072748ded0e852e7d",
                 "shasum": ""
             },
             "require": {
@@ -1574,9 +1784,89 @@
             "support": {
                 "forum": "https://github.com/smarty-php/smarty/discussions",
                 "issues": "https://github.com/smarty-php/smarty/issues",
-                "source": "https://github.com/smarty-php/smarty/tree/v4.2.1"
+                "source": "https://github.com/smarty-php/smarty/tree/v4.3.0"
             },
-            "time": "2022-09-14T10:59:01+00:00"
+            "time": "2022-11-22T21:47:32+00:00"
+        },
+        {
+            "name": "spomky-labs/otphp",
+            "version": "11.1.0",
+            "source": {
+                "type": "git",
+                "url": "https://github.com/Spomky-Labs/otphp.git",
+                "reference": "4849ac1aa560bfc56c0d1534b0d72532da4665ab"
+            },
+            "dist": {
+                "type": "zip",
+                "url": "https://api.github.com/repos/Spomky-Labs/otphp/zipball/4849ac1aa560bfc56c0d1534b0d72532da4665ab",
+                "reference": "4849ac1aa560bfc56c0d1534b0d72532da4665ab",
+                "shasum": ""
+            },
+            "require": {
+                "ext-mbstring": "*",
+                "paragonie/constant_time_encoding": "^2.0",
+                "php": "^8.1"
+            },
+            "require-dev": {
+                "ekino/phpstan-banned-code": "^1.0",
+                "infection/infection": "^0.26",
+                "php-parallel-lint/php-parallel-lint": "^1.3",
+                "phpstan/phpstan": "^1.0",
+                "phpstan/phpstan-deprecation-rules": "^1.0",
+                "phpstan/phpstan-phpunit": "^1.0",
+                "phpstan/phpstan-strict-rules": "^1.0",
+                "phpunit/phpunit": "^9.5.26",
+                "qossmic/deptrac-shim": "^1.0",
+                "rector/rector": "^0.14",
+                "symfony/phpunit-bridge": "^6.1",
+                "symplify/easy-coding-standard": "^11.0"
+            },
+            "type": "library",
+            "autoload": {
+                "psr-4": {
+                    "OTPHP\\": "src/"
+                }
+            },
+            "notification-url": "https://packagist.org/downloads/",
+            "license": [
+                "MIT"
+            ],
+            "authors": [
+                {
+                    "name": "Florent Morselli",
+                    "homepage": "https://github.com/Spomky"
+                },
+                {
+                    "name": "All contributors",
+                    "homepage": "https://github.com/Spomky-Labs/otphp/contributors"
+                }
+            ],
+            "description": "A PHP library for generating one time passwords according to RFC 4226 (HOTP Algorithm) and the RFC 6238 (TOTP Algorithm) and compatible with Google Authenticator",
+            "homepage": "https://github.com/Spomky-Labs/otphp",
+            "keywords": [
+                "FreeOTP",
+                "RFC 4226",
+                "RFC 6238",
+                "google authenticator",
+                "hotp",
+                "otp",
+                "totp"
+            ],
+            "support": {
+                "issues": "https://github.com/Spomky-Labs/otphp/issues",
+                "source": "https://github.com/Spomky-Labs/otphp/tree/11.1.0"
+            },
+            "funding": [
+                {
+                    "url": "https://github.com/Spomky",
+                    "type": "github"
+                },
+                {
+                    "url": "https://www.patreon.com/FlorentMorselli",
+                    "type": "patreon"
+                }
+            ],
+            "time": "2022-11-11T12:57:17+00:00"
         },
         {
             "name": "symfony/polyfill-php81",
@@ -1659,16 +1949,16 @@
         },
         {
             "name": "twbs/bootstrap",
-            "version": "v5.2.2",
+            "version": "v5.3.0-alpha3",
             "source": {
                 "type": "git",
                 "url": "https://github.com/twbs/bootstrap.git",
-                "reference": "961d5ff9844372a4e294980c667bbe7e0651cdeb"
+                "reference": "ac576614a5515e429f27e756fad81d5aa05e95a6"
             },
             "dist": {
                 "type": "zip",
-                "url": "https://api.github.com/repos/twbs/bootstrap/zipball/961d5ff9844372a4e294980c667bbe7e0651cdeb",
-                "reference": "961d5ff9844372a4e294980c667bbe7e0651cdeb",
+                "url": "https://api.github.com/repos/twbs/bootstrap/zipball/ac576614a5515e429f27e756fad81d5aa05e95a6",
+                "reference": "ac576614a5515e429f27e756fad81d5aa05e95a6",
                 "shasum": ""
             },
             "replace": {
@@ -1703,9 +1993,9 @@
             ],
             "support": {
                 "issues": "https://github.com/twbs/bootstrap/issues",
-                "source": "https://github.com/twbs/bootstrap/tree/v5.2.2"
+                "source": "https://github.com/twbs/bootstrap/tree/v5.3.0-alpha3"
             },
-            "time": "2022-10-03T07:44:02+00:00"
+            "time": "2023-04-03T07:26:50+00:00"
         },
         {
             "name": "voku/portable-ascii",
@@ -7095,6 +7385,7 @@
     "aliases": [],
     "minimum-stability": "stable",
     "stability-flags": {
+        "twbs/bootstrap": 15,
         "behat/behat": 0,
         "behat/mink-extension": 0,
         "behat/mink-goutte-driver": 0